Environmental Sound Classification Research Articles

Applying the Generative Model Integrated with the Diffusion Technique to Improve Virtual Sample Generation in Environmental Sound Classification

Applying the Generative Model Integrated with the Diffusion Technique to Improve Virtual Sample Generation in Environmental Sound Classification

Audio feature enhancement based on quaternion filtering and deep hashing

This paper aims to solve the problems of difficult convergence of audio model training, large data demand, and large dimensionality of storage space for audio-generated feature vectors. To this end, this paper proposes the use of quaternion Gabor filtering to suppress the background information of the spectrogram and reduce the interference of the data for the case of insufficient data alignment between audio data and image data after shifting the domain. In addition, different scales of window lengths and frame shifts are used to capture the connections between different vocal objects. To address the problem that the generated feature vectors are large dimensional, we use a deep hash module to map high-dimensional features to low-dimensional features and use a probability function to make the learned samples more consistent with the overall distribution. In the experimental evaluation, the proposed method was evaluated on the environmental sound classification dataset and the music genre classification dataset. The proposed method uses only a common backbone network and improves the accuracy of audio recognition.

Explosion Detection Using Smartphones: Ensemble Learning with the Smartphone High-Explosive Audio Recordings Dataset and the ESC-50 Dataset.

Explosion monitoring is performed by infrasound and seismoacoustic sensor networks that are distributed globally, regionally, and locally. However, these networks are unevenly and sparsely distributed, especially at the local scale, as maintaining and deploying networks is costly. With increasing interest in smaller-yield explosions, the need for more dense networks has increased. To address this issue, we propose using smartphone sensors for explosion detection as they are cost-effective and easy to deploy. Although there are studies using smartphone sensors for explosion detection, the field is still in its infancy and new technologies need to be developed. We applied a machine learning model for explosion detection using smartphone microphones. The data used were from the Smartphone High-explosive Audio Recordings Dataset (SHAReD), a collection of 326 waveforms from 70 high-explosive (HE) events recorded on smartphones, and the ESC-50 dataset, a benchmarking dataset commonly used for environmental sound classification. Two machine learning models were trained and combined into an ensemble model for explosion detection. The resulting ensemble model classified audio signals as either "explosion", "ambient", or "other" with true positive rates (recall) greater than 96% for all three categories.

Environmental sound classification using raw-audio based ensemble framework

Environmental sound classification (ESC) aims to automatically recognize audio recordings from the underlying environment, such as "urban park" or "city centre". Most of the existing methods for ESC use hand-crafted time-frequency features such as log-mel spectrogram to represent audio recordings. However, the hand-crafted features rely on transformations that are defined beforehand and do not consider the variability in the environment due to differences in recording conditions or recording devices. To overcome this, we present an alternative representation framework by leveraging a pre-trained convolutional neural network, SoundNet, trained on a large-scale audio dataset to represent raw audio recordings. We observe that the representations obtained from the intermediate layers of SoundNet lie in low-dimensional subspace. However, the dimensionality of the low-dimensional subspace is not known. To address this, an automatic compact dictionary learning framework is utilized that gives the dimensionality of the underlying subspace. The low-dimensional embeddings are then aggregated in a late-fusion manner in the ensemble framework to incorporate hierarchical information learned at various intermediate layers of SoundNet. We perform experimental evaluation on publicly available DCASE 2017 and 2018 ASC datasets. The proposed ensemble framework improves performance between 1 and 4 percentage points compared to that of existing time-frequency representations.

Environmental sound classification on an embedded hardware platform

Convolutional neural networks (CNNs) have exhibited state-of-the-art performance in various audio classification tasks. However, their real-time deployment remains a challenge on resource constrained devices such as embedded systems. In this paper, we analyze how the performance of large-scale pre-trained audio neural networks designed for audio pattern recognition changes when deployed on a hardware such as a Raspberry Pi. We empirically study the role of CPU temperature, microphone quality and audio signal volume on performance. Our experiments reveal that the continuous CPU usage results in an increased temperature that can trigger an automated slowdown mechanism in the Raspberry Pi, impacting inference latency. The quality of a microphone, specifically with affordable devices such as the Google AIY Voice Kit, and audio signal volume, all affect the system performance. In the course of our investigation, we encounter substantial complications linked to library compatibility and the unique processor architecture requirements of the Raspberry Pi, making the process less straightforward compared to conventional computers (PCs). Our observations, while presenting challenges, pave the way for future researchers to develop more compact machine learning models, design heat-dissipative hardware, and select appropriate microphones when AI models are deployed for real-time applications on edge devices.

HM–GDM: Hybrid Measures and Graph-Dependent Modeling for Environmental Sound Classification

One of the upcoming areas of research is the automated environmental sound classification, as its application is highly utilized in criminal investigations scenarios, surveillance systems, biomedical fields, radio navigation systems, etc. The procedure of environmental sound classification deals with standardized techniques such as feature extraction and feature selection, followed by classification with machine learning or deep learning techniques. In this work, five feature extraction techniques are used initially such as the discrete wavelet transform (DWT)-based empirical mode decomposition (EMD) technique, phase space reconstruction (PSR), kernel principal component analysis (KPCA), variational mode decomposition (VMD) and tunable Q-wavelet transform (TQWT). The extracted features are then selected through seven feature selection techniques, out of which two have been proposed newly and five have already existed. The proposed feature selection techniques are stratified clustered collective technique (SCCT) and fuzzy-based template shape clustering (FTSC). The other feature selection techniques used are the generalized discriminant analysis (GDA), ReliefF, Fisher discriminant criterion (FDC), Kruskal–Wallis test and coati optimization algorithm (COA). The selected features are then classified through the proposed swarm intelligence-based hybrid Adaboost – random forest termed as SIHAR classifier and the selected features are classified with the less explored sparse representation classifier (SRC) along with eight other traditionally used machine learning classifiers. The work also proposes graph-dependent modeling for the environmental sounds where the rhythms are extracted for the similarity assessment and later the decision-making is done using hypothesis testing. The work is tested on Firat ESC-50 dataset and the best results are produced in terms of a high classification accuracy of 87.48% which is obtained for the TQWT + SCCT + SIHAR combination.

Signal latent subspace: A new representation for environmental sound classification

In this study, we propose Signal Latent Subspace (SLS), a flexible method that classifies environmental sound events using the subspace representations of latent features obtained from various neural network-based models. Our main goal is to leverage the high expressiveness of neural networks while retaining the advantages of subspace representation, such as its robustness to noise and ability to work under small sample size (SSS) conditions. We also propose an ensemble strategy native to the subspace representation, to achieve increased performance and reduce the generalization error. We do this through product Grassmann manifold (PGM), resulting in SLS-PGM. Each subspace constructed from latent features of a network can be seen as a point on a factor Grassmann manifold (GM) of a neural network; through PGM, it is possible to unify factor manifolds into a singular representation, and perform classification through a similarity metric on the manifold. We further improve SLS and SLS-PGM in two ways: (1) by using generalized difference subspace (GDS) projection to address the lack of between-class discrimination of subspace representation and (2) by leveraging finetuning regimes to better adapt neural network models to the ESC task. We evaluate our proposed methods, factoring various neural networks, on ESC-10, ESC-50 and UrbanSound environmental sound datasets, and provide extensive ablation experiments and notes for practical use.

ESC-NAS: Environment Sound Classification Using Hardware-Aware Neural Architecture Search for the Edge

The combination of deep-learning and IoT plays a significant role in modern smart solutions, providing the capability of handling task-specific real-time offline operations with improved accuracy and minimised resource consumption. This study provides a novel hardware-aware neural architecture search approach called ESC-NAS, to design and develop deep convolutional neural network architectures specifically tailored for handling raw audio inputs in environmental sound classification applications under limited computational resources. The ESC-NAS process consists of a novel cell-based neural architecture search space built with 2D convolution, batch normalization, and max pooling layers, and capable of extracting features from raw audio. A black-box Bayesian optimization search strategy explores the search space and the resulting model architectures are evaluated through hardware simulation. The models obtained from the ESC-NAS process achieved the optimal trade-off between model performance and resource consumption compared to the existing literature. The ESC-NAS models achieved accuracies of 85.78%, 81.25%, 96.25%, and 81.0% for the FSC22, UrbanSound8K, ESC-10, and ESC-50 datasets, respectively, with optimal model sizes and parameter counts for edge deployment.

Training environmental sound classification models for real-world deployment in edge devices

The interest in smart city technologies has grown in recent years, and a major challenge is to develop methods that can extract useful information from data collected by sensors in the city. One possible scenario is the use of sound sensors to detect passing vehicles, sirens, and other sounds on the streets. However, classifying sounds in a street environment is a complex task due to various factors that can affect sound quality, such as weather, traffic volume, and microphone quality. This paper presents a deep learning model for multi-label sound classification that can be deployed in the real world on edge devices. We describe two key components, namely data collection and preparation, and the methodology to train the model including a pre-train using knowledge distillation. We benchmark our models on the ESC-50 dataset and show an accuracy of 85.4%, comparable to similar state-of-the-art models requiring significantly more computational resources. We also evaluated the model using data collected in the real world by early prototypes of luminaires integrating edge devices, with results showing that the approach works well for most vehicles but has significant limitations for the classes “person” and “bicycle”. Given the difference between the benchmarking and the real-world results, we claim that the quality and quantity of public and private data for this type of task is the main limitation. Finally, all results show great benefits in pretraining the model using knowledge distillation.

Audio Scanning Network: Bridging Time and Frequency Domains for Audio Classification

With the rapid growth of audio data, there's a pressing need for automatic audio classification. As a type of time-series data, audio exhibits waveform fluctuations in both the time and frequency domains that evolve over time, with similar instances sharing consistent patterns. This study introduces the Audio Scanning Network (ASNet), designed to leverage abundant information for achieving stable and effective audio classification. ASNet captures real-time changes in audio waveforms across both time and frequency domains through reservoir computing, supported by Reservoir Kernel Canonical Correlation Analysis (RKCCA) to explore correlations between time-domain and frequency-domain waveform fluctuations. This innovative approach empowers ASNet to comprehensively capture the changes and inherent correlations within the audio waveform, and without the need for time-consuming iterative training. Instead of converting audio into spectrograms, ASNet directly utilizes audio feature sequences to uncover associations between time and frequency fluctuations. Experiments on environmental sound and music genre classification tasks demonstrate ASNet's comparable performance to state-of-the-art methods.

A Lightweight Channel and Time Attention Enhanced 1D CNN Model for Environmental Sound Classification

One dimension convolutional neural networks (1D CNN) that directly take raw waveforms as input has less competition than 2D CNN recognizing environmental sound. In order to overcome its disadvantages, we propose a novel lightweight 1D CNN structure by employing attention mechanism, which has significant improvement in both accuracy and computational complexity. Concretely, (1) two attention modules are constructed along channel and time dimension separately, and combined to give an intermediate feature map, which focus on key frequency band and semantically related time frame information. (2) Without increasing training overhead, snapshot ensemble is employed to further improve performance. Results from two benchmarking datasets (UrbanSound8k, ESC-10) demonstrated that: by employing attention mechanism, our model outperforms all of the previously reported 1D CNN approaches in accuracy with less parameters. Meanwhile with improved performance gain, the proposed model is superior than most of the existing spectral-based 2D CNN approaches and competitive with SOTA performance, while with orders of magnitude parameters fewer. Overall, it indicates our model is compact and has good potential in practical resource-limited applications, such as sound recognition on embedded platform.

Multiclass environmental sound classification model based on adding residual connections to self-attention layers

Multiclass environmental sound classification model based on adding residual connections to self-attention layers

Data-centric Methods for Environmental Sound Classification with Limited Labels

Data-centric Methods for Environmental Sound Classification with Limited Labels

Robust technique for environmental sound classification using convolutional recurrent neural network

Robust technique for environmental sound classification using convolutional recurrent neural network

Concatenation-based pre-trained convolutional neural networks using attention mechanism for environmental sound classification

Environmental sound classification (ESC) is gaining popularity in the field of information processing due to its significance in non-speech audio categorization. ESC faces challenges in categorizing ambient sounds, which lack a clear structure, unlike speech or music sounds. Deep learning (DL) techniques are widely used in ESC to extract relevant information from ambient sounds, in which feature extraction is very crucial and directly impacts the classification performance. However, feature extraction can be computationally expensive, especially when employing complex non-linear techniques, due to the significant amount of computing resources required during the training process of DL models. Moreover, environmental sounds can exhibit significant variability in terms of their temporal and spectral characteristics, which can pose challenges to training DL models effectively. In order to overcome these limitations, this research proposes an efficient method for extracting meaningful features from audio files and improving DL techniques' performance using spectrogram images generated from sound environmental datasets. The proposed approach uses convolutional neural networks (CNNs) with attention mechanisms and appropriate data augmentation methods. Unlike pre-trained models that use a single vector for feature extraction, the proposed approach uses a new concatenation-based CNN model with attention mechanisms, which can more effectively capture intricate relationships between input data features. This approach allows for the extraction of features from different sectors of the feature space, resulting in a more precise classification of complex and diverse datasets. Additionally, the proposed approach leverages the parallel extraction feature technique to extract features from multiple CNN models, which improves classification performance. Furthermore, attention modules are used to focus on the most relevant features of the input data. Comparative experiments are also conducted for the proposed approach and some existing state-of-the-art methods, and the results show that the former has better classification performance.

A High Accuracy and Low Power CNN-Based Environmental Sound Classification Processor

A High Accuracy and Low Power CNN-Based Environmental Sound Classification Processor

Predictions for sound events and soundscape impressions from environmental sound using deep neural networks

In this study, we investigate methods for quantifying soundscape impressions, meaning pleasantness and eventfulness, from environmental sounds. From a point of view of Machine Learning (ML) research areas, acoustic scene classification (ASC) tasks and sound event classification (SEC) tasks are intensively studied and their results are helpful to consider soundscape impressions. In general, while most of ASCs and SECs use only sound for the classifications, a soundscape impression should not be perceived just from a sound but perceived from a sound with a landscape by human beings. Therefore, to establish automatic quantification of soundscape impressions, it should use other information such as landscape in addition to sound. First, we tackle to predict of two soundscape impressions using sound data collected by the cloud sensing method. For this purpose, we have proposed prediction method of soundscape impressions using environmental sounds and aerial photographs. Second, we also tackle to establish environmental sound classification using feature extractor trained by Variational Autoencoder (VAE). The feature extractor by VAE can be trained as unsupervised learning, therefore, it could be promising approaches for the growth dataset like as our cloud-sensing data collection schemes. Finally, we discuss about an integration for these methods.

A Survey on Deep Learning Based Forest Environment Sound Classification at the Edge

Forest ecosystems are of paramount importance to the sustainable existence of life on earth. Unique natural and artificial phenomena pose severe threats to the perseverance of such ecosystems. With the advancement of artificial intelligence technologies, the effectiveness of implementing forest monitoring systems based on acoustic surveillance has been established due to the practicality of the approach. It can be identified that with the support of transfer learning, deep learning algorithms outperform conventional machine learning algorithms for forest acoustic classification. Further, a clear requirement to move the conventional cloud-based sound classification to the edge is raised among the research community to ensure real-time identification of acoustic incidents. This article presents a comprehensive survey on the state-of-the-art forest sound classification approaches, publicly available datasets for forest acoustics, and the associated infrastructure. Further, we discuss the open challenges and future research aspects that govern forest acoustic classification.

Urban Noise Analysis and Emergency Detection System using Lightweight End-to-End Convolutional Neural Network

In recent years, the application of deep learning to environmental sound classification (ESC) has received considerable attention owing to its powerful ability to recognize the context of urban sounds. In general, deep learning models with high accuracy require substantial computing and memory resources. Consequently, to apply complex deep learning models to ESC in the real world, model inference has been performed on cloud servers with powerful computing resources. However, heavy network traffic and security issues occur when inferences are performed on a cloud server. In addition, deploying a deep learning model trained on a single public ESC dataset may not be sufficient for classifying various classes of urban noise and emergency-related sounds. To address these problems, we propose an on-device, real-time urban sound monitoring system that can classify various urban sounds at low system construction costs. The proposed system consisted of an edge artificial intelligence (AI) node and a FIWARE-based server. To enable the real-time inference on a resource-constrained edge AI node, we developed a lightweight convolutional neural network (CNN) by adjusting the input and model configurations to achieve high accuracy with a low number of parameters. The model achieved 94.9% classification accuracy using only 331 K parameters on an integrated dataset that included 17 classes of urban noises and emergencies. Furthermore, a prototype of the proposed system was developed and evaluated to verify its feasibility. The prototype system was built at a cost of less than USD 50 and could perform the entire monitoring process every 3 s. We also visualized the sound monitoring data using Grafana on a FIWARE-based server.

MSARN: A Multi-scale Attention Residual Network for End-to-End Environmental Sound Classification

MSARN: A Multi-scale Attention Residual Network for End-to-End Environmental Sound Classification