Environmental Sound Classification Research Articles

Environmental Sound Classification Based on Attention Feature Fusion and Improved Residual Network

Environmental Sound Classification Based on Attention Feature Fusion and Improved Residual Network

An Automatic Classification System for Environmental Sound in Smart Cities.

With the continuous promotion of "smart cities" worldwide, the approach to be used in combining smart cities with modern advanced technologies (Internet of Things, cloud computing, artificial intelligence) has become a hot topic. However, due to the non-stationary nature of environmental sound and the interference of urban noise, it is challenging to fully extract features from the model with a single input and achieve ideal classification results, even with deep learning methods. To improve the recognition accuracy of ESC (environmental sound classification), we propose a dual-branch residual network (dual-resnet) based on feature fusion. Furthermore, in terms of data pre-processing, a loop-padding method is proposed to patch shorter data, enabling it to obtain more useful information. At the same time, in order to prevent the occurrence of overfitting, we use the time-frequency data enhancement method to expand the dataset. After uniform pre-processing of all the original audio, the dual-branch residual network automatically extracts the frequency domain features of the log-Mel spectrogram and log-spectrogram. Then, the two different audio features are fused to make the representation of the audio features more comprehensive. The experimental results show that compared with other models, the classification accuracy of the UrbanSound8k dataset has been improved to different degrees.

BSN-ESC: A Big-Small Network-Based Environmental Sound Classification Method for AIoT Applications.

In recent years, environmental sound classification (ESC) has prevailed in many artificial intelligence Internet of Things (AIoT) applications, as environmental sound contains a wealth of information that can be used to detect particular events. However, existing ESC methods have high computational complexity and are not suitable for deployment on AIoT devices with constrained computing resources. Therefore, it is of great importance to propose a model with both high classification accuracy and low computational complexity. In this work, a new ESC method named BSN-ESC is proposed, including a big-small network-based ESC model that can assess the classification difficulty level and adaptively activate a big or small network for classification as well as a pre-classification processing technique with logmel spectrogram refining, which prevents distortion in the frequency-domain characteristics of the sound clip at the joint part of two adjacent sound clips. With the proposed methods, the computational complexity is significantly reduced, while the classification accuracy is still high. The proposed BSN-ESC model is implemented on both CPU and FPGA to evaluate its performance on both PC and embedded systems with the dataset ESC-50, which is the most commonly used dataset. The proposed BSN-ESC model achieves the lowest computational complexity with the number of floating-point operations (FLOPs) of only 0.123G, which represents a reduction of up to 2309 times in computational complexity compared with state-of-the-art methods while delivering a high classification accuracy of 89.25%. This work can achieve the realization of ESC being applied to AIoT devices with constrained computational resources.

When Deep is not Enough: Towards Understanding Shallow and Continual Learning Models in Realistic Environmental Sound Classification for Robots

Although deep learning models are state-of-the-art models in audio classification, they fall short when applied in developmental robotic settings and human–robot interaction (HRI). The major drawback is that deep learning relies on supervised training with a large amount of data and annotations. In contrast, developmental learning strategies in human–robot interaction often deal with small-scale data acquired from HRI experiments and require the incremental addition of novel classes. Alternatively, shallow learning architectures that enable fast and yet robust learning are provided by simple distance metric-based learning and neural architectures implementing the reservoir computing paradigm. Similarly, continual learning algorithms receive more attention in the last years as they can integrate stable perceptual feature extraction using pre-trained deep learning models with open-set classification. As our research centers around reenacting the incremental learning of audio cues, we conducted a study on environmental sound classification using the iCaRL as well as the GDumb continual learning algorithms in comparison with a popular classifier in this domain, the kNN classifier, as well as employing an Echo State Network. We contrast our results with those obtained from a VGGish network that serves here as the performance upper bound that allows us to quantify the performance differences and to discuss current issues with continual learning in the audio domain. As only little is known about using shallow models or continual learning in the audio domain, we pass on additional techniques like data augmentation and create a simple experimental pipeline that is easy to reproduce. Although our selected algorithms are partially inferior in performance compared to the upper bound, our evaluation on three environmental sound datasets shows promising performance using continual learning for a subset of the DCASE2019 challenge dataset and the ESC10 dataset. As we do not address benchmarking in this paper, our study provides a good foundation for further research and computational improvements on shallow and continual learning models for robotic applications in the audio domain.

Environmental sound classification based on improved compact bilinear attention network

Aiming at the difficulty in effectively extracting representative features from environmental sounds characterized by the complicated and considerable variation in the time-frequency domain characteristics, an improved compact bilinear attention network (ICBA-Net) is proposed to more precisely focus on the regions that need attention in feature maps in this paper. Firstly, log-mel spectrogram (LMS) extracted from environmental sound is fed into two parallel pre-trained densely connected convolutional network (DenseNet) for deep feature extraction. Afterwards, a coordinate attention module (CAM) is utilized in each branch to integrate key information in both horizontal and vertical directions to achieve a more accurate focus on global feature salient regions. Finally, two-stream enhanced global representations are integrated into a compact bilinear vector, followed by the full connection layer with activation function softmax to output the final prediction result. Experiments on two publicly available datasets (ESC-10 and ESC-50) and some environmental sounds collected from common scenarios in our real life are used to verify the validity of the proposed method. Through comparison and verification, the results show that the proposed network outperformed almost all state-of-the-art systems regarding classification accuracy, and the feasibility and effectiveness of CAM have also been verified.

Performance Evaluation of Environmental Sound Classification: A Machine Learning Stacking and Multi-Criteria Metrics Based Approach

This study proposes an Environment Sound Classification Task (ESC) model that includes numerous element channels given as a contribution to Machine learning with an Attention instrument. ESC is a significant testing issue. The interest in the paper lies in utilizing different part channels involving the MFCCs-Mel Frequency Cepstral Coefficients a mutual module in speaker detection and artificial speech systems. LPCs-Linear Prediction Coefficients and Linear Prediction Cepstral Coefficients were the most commonly used types in ASR- Automated speech recognition. The paper also discusses some basic features of MFCCs and how to put them into practice. The techniques used for this project are Background Gaussian Noise and Time Shifting, by observing that every technique is implemented with a provided probability, however, when at the time of generation of a new sample the same spectrogram input may have several combinations. We go through the blend information expansion method to additional lift execution. Our model can accomplish cutting-edge execution on three benchmark climate sound characterization datasets, for example, the UrbanSound8K.

Deep learning based source identification of environmental audio signals using optimized convolutional neural networks

The research in the field of environmental sounds is a growing area due to its enormous potential and its applications. One of the major factors that affect the model performance is the noisy, redundant, or irrelevant features. Deep learning models have shown promise in this area, but the extraction of optimal features from audio signals and classification efficiency of the model are still challenging issues in this field. To address the challenges faced by existing methods, this research proposes a unique deep learning framework-based model that employs an enhanced bio-inspired algorithm for feature extraction and environmental sound classification. The quality and relevance of the training features are essential for the model’s accuracy, and a novel algorithm is introduced to select optimal features for improved performance. The algorithm is further improved for weight optimization to address overfitting and accuracy issues. Additionally, a modified version of the Discrete Fourier Transform is introduced to reduce computational complexity, which makes the model more suitable for real-time applications or resource-limited devices. This research emphasizes the necessity for improved algorithms for feature selection and weight optimization. The proposed model exhibits excellent accuracy and efficiency, making it suitable for real-time applications.

Environmental Sound Classification Framework Based on L-mHP Features and SE-ResNet50 Network Model

Environmental sound classification (ESC) tasks are attracting more and more attention. Due to the complexity of the scene and personnel mobility, there are some difficulties in understanding and generating environmental sound models for ESC tasks. To address these key issues, this paper proposes an audio classification framework based on L-mHP features and the SE-ResNet50 model and improves a dual-channel data enhancement scheme based on a symmetric structure for model training. Firstly, this paper proposes the L-mHP feature to characterize environmental sound. The L-mHP feature is a three-channel feature consisting of a Log-Mel spectrogram, a harmonic spectrogram, and a percussive spectrogram. The harmonic spectrogram and percussive spectrogram can be obtained by harmonic percussive source separation (HPSS) of a Log-Mel spectrogram. Then, an improved audio classification model SE-ResNet50 is proposed based on the ResNet-50 model. In this paper, a dual-channel data enhancement scheme based on a symmetric structure is promoted, which not only makes the audio variants more diversified, but also makes the model focus on learning the time–frequency mode in the acoustic features during the training process, so as to improve the generalization performance of the model. Finally, the audio classification experiment of the framework is carried out on public datasets. An experimental accuracy of 94.92%, 99.67%, and 90.75% was obtained on ESC-50, ESC-10 and UrbanSound8K datasest, respectively. In order to simulate the classification performance of the framework in the actual environment, the framework was also evaluated on a self-made sound dataset with different signal-to-noise ratios. The experimental results show that the proposed audio classification framework has good robustness and feasibility.

환경음 분류와 위치 정보를 이용한 딥러닝 기반 음성 품질 향상 기법

음성 처리 분야에서 딥러닝은 음성인식의 정밀도를 향상시켜 많은 발전을 이루었다. 그중 하나인 음성 향상은 말하는 음성과 잡음이 섞여있는 입력에서 음성과 잡음을 분리시켜 음성 품질을 향상시킬 수 있는 기법이다. 이는 AI-스피커, 스마트폰 등에 활용되어 사람 대 사람, 기계 대 인간과의 의사소통을 원활하게 하고 로봇이나 문자 음성 변환(Text-to-Speech) 등을 위한 깨끗한 음성 데이터 수집을 가능하게 한다. 하지만 단일 모델만을 사용한 기존 음성 향상 기법은 환경마다 특수적으로 발생하는 잡음 제거에 있어 효과적이지 못하다. 환경 특수적으로 발생할 수 있는 잡음을 효과적으로 제거하기 위해, 본 논문은 음향 장면 분류 기법과 위치 정보 활용 기술을 결합하여 최적의 환경별 음성 향상이 이루어 질 수 있게 하는 딥러닝 모델을 제안한다. 실험 결과 SNR 9 dB을 기준으로 하였을 때, 본 기법이 기존 딥러닝 모델대비 PESQ 값은 평균 0.06 이상, STOI 값은 평균 0.015 이상의 향상된 품질을 보임을 확인하였다.

Environmental Sound Classification Based on CAR-Transformer Neural Network Model

Environmental Sound Classification Based on CAR-Transformer Neural Network Model

Using audio content and emotional response to predict soundscape perception through machine learning

This study is concerned with utilizing machine learning techniques for predicting soundscape perception by identifying the audio content of soundscapes and linking it with people's reported emotional responses. This research goal required developing an environmental sound classification model; however, the capabilities of these algorithms have some significant drawbacks. Supervised learning algorithms need a large number of labelled audio samples for each sound category. Given that a model for classifying environmental sound must be trained using a wide range of sound sources, this presents a substantial problem for developing a robust model that generalizes well to different environments. We prepared a convolutional neural network (CNN) based classifier; however, to tackle the limitations, we used musical instruments for the training dataset rather than environmental sound sources and optimized the neural network for this task. Based on how closely the soundscapes' audio content resembled the musical instruments in the dataset, CNN classified the soundscapes' audio content. We then conducted an online soundscape perception survey to evaluate participants' emotional responses to numerous soundscape clips. We prepared a feedforward neural network, which used the sound classification model's audio content output with the survey data to create a model for predicting people's responses to different soundscapes.

A Novel Lightweight Audio-visual Saliency Model for Videos

Audio information has not been considered an important factor in visual attention models regardless of many psychological studies that have shown the importance of audio information in the human visual perception system. Since existing visual attention models only utilize visual information, their performance is limited but also requires high-computational complexity due to the limited information available. To overcome these problems, we propose a lightweight audio-visual saliency (LAVS) model for video sequences. To the best of our knowledge, this article is the first trial to utilize audio cues for an efficient deep-learning model for the video saliency estimation. First, spatial-temporal visual features are extracted by the lightweight receptive field block (RFB) with the bidirectional ConvLSTM units. Then, audio features are extracted by using an improved lightweight environment sound classification model. Subsequently, deep canonical correlation analysis (DCCA) aims at capturing the correspondence between audio and spatial-temporal visual features, thus obtaining a spatial-temporal auditory saliency. Lastly, the spatial-temporal visual and auditory saliency are fused to obtain the audio-visual saliency map. Extensive comparative experiments and ablation studies validate the performance of the LAVS model in terms of effectiveness and complexity.

Episodic memory based continual learning without catastrophic forgetting for environmental sound classification

Episodic memory based continual learning without catastrophic forgetting for environmental sound classification

Forest Sound Classification Dataset: FSC22.

The study of environmental sound classification (ESC) has become popular over the years due to the intricate nature of environmental sounds and the evolution of deep learning (DL) techniques. Forest ESC is one use case of ESC, which has been widely experimented with recently to identify illegal activities inside a forest. However, at present, there is a limitation of public datasets specific to all the possible sounds in a forest environment. Most of the existing experiments have been done using generic environment sound datasets such as ESC-50, U8K, and FSD50K. Importantly, in DL-based sound classification, the lack of quality data can cause misguided information, and the predictions obtained remain questionable. Hence, there is a requirement for a well-defined benchmark forest environment sound dataset. This paper proposes FSC22, which fills the gap of a benchmark dataset for forest environmental sound classification. It includes 2025 sound clips under 27 acoustic classes, which contain possible sounds in a forest environment. We discuss the procedure of dataset preparation and validate it through different baseline sound classification models. Additionally, it provides an analysis of the new dataset compared to other available datasets. Therefore, this dataset can be used by researchers and developers who are working on forest observatory tasks.

인공지능 학습모델과 음향데이터를 이용한 군사장비 식별

Acoustic data is widely used for the development of voice recognition based services, speaker identification system, or environmental sound classification. However, in the defense sector, research on the use of acoustic data and AI learning models has not yet been actively conducted. In addition, collection methods or quality standards that must be considered in order to use acoustic data as training data have not been clearly established. Therefore, this paper attempts to classify six weapons for the first time using four AI learning models and previously collected acoustic data to derive considerations in the process of collecting acoustic data. The learning data used in the experiment is acoustic data related to Army equipment collected through a government-supported project in the second half of 2021. Experimental results show that the classification performance of CNN and LightGBM models is generally high and the best performance is achieved when the data has a medium length. By uniformly re-configuring the number of data per class using SMOTE, the performance can be further improved. Based on these experimental results, we also propose what should be considered when constructing AI training data for classification of military equipment based on acoustic data.

Divide and Distill: New Outlooks on Knowledge Distillation for Environmental Sound Classification

Environmental sound classification (ESC) is an important research problem with a broad range of applications including audio-based surveillance, audio-visual systems, smart homes, and robotics, among others. The recently proposed vision multi-layer perceptron-mixer (MLP-mixer) has outperformed traditional deep models (CNN or ResNet) and attained new state-of-the-art performances for several computer vision applications (image/video classification and image segmentation). Following the success of MLP-mixer, in this paper, we propose a novel audio MLP-mixer (AMM) network that classifies the different types of environmental sounds. Despite the higher performance, the high computational cost (number of trainable parameters and floating point operations) prohibits deployment of the AMM model on edge for designing real-life applications. To alleviate the aforementioned issue, in this work, we present three different knowledge distillation (KD) strategies to train a compact deep network for ESC. The proposed strategies divide the input Mel-spectrogram into patches and a lightweight deep ESC model is trained in the presence of three teacher networks under the offline KD training framework. Additionally, we have designed two novel loss functions for KD that are free from a temperature parameter that need to be set manually by a user as in the case of the traditional vanilla KD technique. We conducted our experiments on three benchmark ESC datasets namely ESC-10, Urbansound8k (US8K), and DCASE-2019 Task-1(A). The obtained results demonstrate the significance of utilization of proposed methods over other existing KD methods in terms of classification accuracy.

Environmental Sound Classification With Low-Complexity Convolutional Neural Network Empowered by Sparse Salient Region Pooling

Environmental Sound Classification (ESC) is an important field in a broad range of applications, such as smart cities, audio surveillance, and health care. Recently, Convolutional Neural Networks (CNNs) have taken the lead from traditional approaches and have produced promising results. However, the achieved improvements are often accompanied by increasing depth, complexity, and size of the network, which prevents their usage in many practical applications. In this work, our goal is to empower a small-size low-complexity CNN model to achieve superior performance. To this end, we concentrate on the importance of global pooling technique, which is less investigated in ESC. In most previous works, models utilize global average pooling layer which does not consider regional saliency, and thus weakens the salient time-frequency regions contributions to the classification, and also to the training of convolutional kernels. We propose a novel global pooling method, called Sparse Salient Region Pooling (SSRP), which computes the channel descriptors using a sparse subset of features, and guides the model to effectively learn from the more salient time-frequency regions. Experimental results demonstrate that the proposed model with only 700K parameters yields accuracies of 86.7% on ESC-50 and 94.8% on ESC-10, which are comparable to that of the state-of-the-art methods. Compared to the baseline model, our model achieves absolute improvement of 21.8% in accuracy on ESC-50, with 98% smaller model size. Our visual analyses show that SSRP intensifies the responses of low-energy regions such that they contribute even more than high-energy regions to the classification of specific sound classes.

QUESC: Environmental Sound classification Using Quantum Quantized Networks

Environmental Sound classification (ESC) is crucial in various fields such as audio surveillance, wildlife conservation, and acoustic monitoring. However, traditional methods often struggle to handle the curse of dimensionality and the limitations of classical machine learning algorithms. In the paper, a novel approach is proposed for ESC using a Quantized Hybrid Quantum-Classical Neural Network (QQNN), which leverages the potential advantages of quantum computing. The Hybrid QQNN architecture aims to reduce the required parameters and achieve performance comparable to traditional methods. It uses a set of pre-trained layers to reduce the number of features for the parameterized quantum circuit (PQC). This approach was evaluated using the ESC-10 dataset in Pennylane's environment to train the hybrid quantum model. The results demonstrate that the proposed Hybrid QQNN approach achieves comparable accuracy with the classical machine learning methods. This opens up the possibility of performing additional transfer learning in the future.

Comparative Analysis of Different Sampling Rates on Environmental Sound Classification Using the Urbansound8k Dataset

Environmental sound classification (ESC) has gained increasing attention in recent years. This study focuses on the evaluation of the popular public dataset Urbansound8k (Us8k) at different sampling rates using hand crafted features. The Us8k dataset contains environment sounds recorded at various sampling rates, and previous ESC works have uniformly resampled the dataset. Some previous work converted this data to different sampling rates for various reasons. Some of them chose to convert the rest of the dataset to 44,100, as the majority of the Us8k files were already at that sampling rate. On the other hand, some researchers down sampled the dataset to 8000, as it reduced computational complexity, while others resampled it to 16,000, aiming to achieve a balance between higher classification accuracy and lower computational complexity. In this research, we assessed the performance of ESC tasks using sampling rates of 8000 Hz, 16,000 Hz, and 44,100 Hz by extracting the hand crafted features Mel frequency cepstral coefficient (MFCC), gamma tone cepstral coefficients (GTCC), and Mel Spectrogram (MelSpec). The results indicated that there was no significant difference in the classification accuracy among the three tested sampling rates.

Environmental Sound Classification using Hybrid Ensemble Model

Environmental sound classification (ESC) is the most trending research areas. The sounds in the surroundings such as screaming, air conditioners, and rain droplets can help in the development of context-aware applications. It is complex to process the envi- ronmental sounds as compared to speech and music due to the unstructured essence of environmental sounds. In the past, certain preprocessing techniques, feature extraction, and classification algorithms are used for ESC. Several researchers have applied ma- chine learning classifiers for ESC and certain ensemble classifiers are also used but the accuracy can be increased if instead of combining homogeneous classifiers, heterogeneous classifiers can be ensembled. In this paper, a hybrid ensemble classifier is used for ESC on the UrbanSound8k dataset and cepstral features Mel Frequency Cepstral Coefficients are used. Five different machine learning classifiers- Decision Tree, Support Vector Machine, Logistic Regression, K- Nearest Neighbour, and Naive Bayes are used to develop a hybrid ensemble model. The highest accuracy is obtained when all the five classifiers are combined. The proposed approach gives an accuracy of 79.4% and is compared with the benchmark results using individual classifiers and the former out- performs the latter. The results of the hybrid ensemble model on the UrbanSound8K dataset are also compared with the dataset ESC-10.