Environmental Sound Classification Research Articles

Environmental Sound Classification: A descriptive review of the literature

• A descriptive survey for Environmental sound classification (ESC) detailing datasets, preprocessing techniques, features and classifiers is done. • No recent survey related to ESC is published. • The potential benefits of ESC are development of hearing aids, crime investigation and security systems. Automatic environmental sound classification (ESC) is one of the upcoming areas of research as most of the traditional studies are focused on speech and music signals. Classifying environmental sounds such as glass breaking, helicopter, baby crying and many more can aid in surveillance systems as well as criminal investigations. In this paper, a vast range of literature in the field of ESC is elucidated from various facets like preprocessing, feature extraction, and classification techniques. Researchers have used various noise removal and signal enhancement techniques to preprocess the signals. This paper explicates multitude of datasets used in recent studies along with the year of publication and maximum accuracy achieved with the dataset. Deep Neural Networks surpass the traditional machine learning classifiers. The future challenges and prospective research in this field are proposed. Since no recent review on ESC has been published, this study will open up novel ways for certain business applications and security systems.

Attention Based Convolutional Neural Network with Multi-frequency Resolution Feature for Environment Sound Classification.

The environmental sound classification has great research significance in the fields of intelligent audio monitoring and other fields. A novel multi-frequency resolution (MFR) feature is proposed in this paper to solve the problem that the existing single frequency resolution time–frequency features of sound cannot effectively express the characteristics of multiple types of sound. The MFR feature is composed of three features with different frequency resolutions, which are compressed in varying degrees at the time dimension. This method not only has the effect of data augmentation but also can obtain more context information during the feature extraction. And the MFR features of Log-Mel Spectrogram, Cochleagram, and Constant Q-Transform are combined to form a multi-channel MFR feature. Also, a network named SacNet is built, which can effectively solve the problem that the time–frequency feature map of sound contains more invalid information. The basic structural unit of the SacNet consists of two parallel branches, one using depthwise separable convolution as the main feature extractor, and the other using spatial attention module to extract more effective information. Experiment results have demonstrated that the proposed method achieves the state-of-the-art accuracy of 97.5%, 93.1%, and 95.3% on three benchmark datasets of ESC10, ESC50, and UrbanSound8K respectively, which are increased by 3.3%, 0.5%, and 2.3% respectively compared with the previous advanced methods.

Vehicle Interior Sound Classification Based on Local Quintet Magnitude Pattern and Iterative Neighborhood Component Analysis

ABSTRACT Nowadays, environmental sound classification (ESC) has become one of the most studied research areas. Sound signals that are indistinguishable from the human auditory systems have been classified by computer-aided systems and machine learning methods. Therefore, ESC has been used in signal processing and sound forensics applications. A novel ESC type is presented in this paper, and it is named as vehicle interior sound classification (VISC). VISC is defined as one of the sub-branches of the ESC, and it is utilized as sound-based biometrics for vehicles. A hand-crafted feature-based VISC method is presented. The proposed method has multileveled feature generation by using maximum pooling and the proposed local quintet magnitude pattern (LQMP), feature selection with iterative neighborhood component analysis (INCA), and classification phases. A novel VISC dataset was collected from YouTube and the proposed LQMP and INCA based method applied to the collected sounds. The results denoted that following: the accuracy, F1-score, and geometric mean of the proposed LQMP and INCA based VISC method were calculated as 98.38%,98.23%, and 98.21% by using support vector machine classifier respectively. The contribution of the proposed VISC method is to denote that the vehicles can be classified by using sound.

Empirical Mode Decomposition-Based Feature Extraction for Environmental Sound Classification

In environment sound classification, log Mel band energies (MBEs) are considered as the most successful and commonly used features for classification. The underlying algorithm, fast Fourier transform (FFT), is valid under certain restrictions. In this study, we address these limitations of Fourier transform and propose a new method to extract log Mel band energies using amplitude modulation and frequency modulation. We present a comparative study between traditionally used log Mel band energy features extracted by Fourier transform and log Mel band energy features extracted by our new approach. This approach is based on extracting log Mel band energies from estimation of instantaneous frequency (IF) and instantaneous amplitude (IA), which are used to construct a spectrogram. The estimation of IA and IF is made by associating empirical mode decomposition (EMD) with the Teager–Kaiser energy operator (TKEO) and the discrete energy separation algorithm. Later, Mel filter bank is applied to the estimated spectrogram to generate EMD-TKEO-based MBEs, or simply, EMD-MBEs. In addition, we employ the EMD method to remove signal trends from the original signal and generate another type of MBE, called S-MBEs, using FFT and a Mel filter bank. Four different datasets were utilised and convolutional neural networks (CNN) were trained using features extracted from Fourier transform-based MBEs (FFT-MBEs), EMD-MBEs, and S-MBEs. In addition, CNNs were trained with an aggregation of all three feature extraction techniques and a combination of FFT-MBEs and EMD-MBEs. Individually, FFT-MBEs achieved higher accuracy compared to EMD-MBEs and S-MBEs. In general, the system trained with the combination of all three features performed slightly better compared to the system trained with the three features separately.

Fractional-Order Calculus-Based Data Augmentation Methods for Environmental Sound Classification with Deep Learning

In this paper, we propose two fractional-order calculus-based data augmentation methods for audio signals. The first approach is based on fractional differentiation of the Mel scale. By using a randomly selected fractional derivation order, we are warping the Mel scale, therefore, we aim to augment Mel-scale-based time-frequency representations of audio data. The second approach is based on previous fractional-order image edge enhancement methods. Since multiple deep learning approaches treat Mel spectrogram representations like images, a fractional-order differential-based mask is employed. The mask parameters are produced with respect to randomly selected fractional-order derivative parameters. The proposed data augmentation methods are applied to the UrbanSound8k environmental sound dataset. For the classification of the dataset and testing the methods, an arbitrary convolutional neural network is implemented. Our results show that fractional-order calculus-based methods can be employed as data augmentation methods. Increasing the dataset size to six times the original size, the classification accuracy result increased by around 8.5%. Additional tests on more complex networks also produced better accuracy results compared to a non-augmented dataset. To our knowledge, this paper is the first example of employing fractional-order calculus as an audio data augmentation tool.

Comparison of semi-supervised deep learning algorithms for audio classification

In this article, we adapted five recent SSL methods to the task of audio classification. The first two methods, namely Deep Co-Training (DCT) and Mean Teacher (MT), involve two collaborative neural networks. The three other algorithms, called MixMatch (MM), ReMixMatch (RMM), and FixMatch (FM), are single-model methods that rely primarily on data augmentation strategies. Using the Wide-ResNet-28-2 architecture in all our experiments, 10% of labeled data and the remaining 90% as unlabeled data for training, we first compare the error rates of the five methods on three standard benchmark audio datasets: Environmental Sound Classification (ESC-10), UrbanSound8K (UBS8K), and Google Speech Commands (GSC). In all but one cases, MM, RMM, and FM outperformed MT and DCT significantly, MM and RMM being the best methods in most experiments. On UBS8K and GSC, MM achieved 18.02% and 3.25% error rate (ER), respectively, outperforming models trained with 100% of the available labeled data, which reached 23.29% and 4.94%, respectively. RMM achieved the best results on ESC-10 (12.00% ER), followed by FM which reached 13.33%. Second, we explored adding the mixup augmentation, used in MM and RMM, to DCT, MT, and FM. In almost all cases, mixup brought consistent gains. For instance, on GSC, FM reached 4.44% and 3.31% ER without and with mixup. Our PyTorch code will be made available upon paper acceptance at https://github.com/Labbeti/SSLH.

Environmental Sound Classification on the Edge: A Pipeline for Deep Acoustic Networks on Extremely Resource-Constrained Devices

Significant efforts are being invested to bring state-of-the-art classification and recognition to edge devices with extreme resource constraints (memory, speed, and lack of GPU support). Here, we demonstrate the first deep network for acoustic recognition that is small, flexible and compression-friendly yet achieves state-of-the-art performance for raw audio classification. Rather than handcrafting a once-off solution, we present a generic pipeline that automatically converts a large deep convolutional network via compression and quantization into a network for resource-impoverished edge devices. After introducing ACDNet, which produces above state-of-the-art accuracy on ESC-10 (96.65%), ESC-50 (87.10%), UrbanSound8K (84.45%) and AudioEvent (92.57%), we describe the compression pipeline and show that it allows us to achieve 97.22% size reduction and 97.28% FLOP reduction while maintaining close to state-of-the-art accuracy 96.25%, 83.65%, 78.27% and 89.69% on these datasets. We describe a successful implementation on a standard off-the-shelf microcontroller and, beyond laboratory benchmarks, report successful tests on real-world datasets.

Case study: Empirical study on indoor environmental sound classification to support hearing-impaired persons

Hearing-impaired persons often find it extremely difficult to catch and identify environmental sounds, which often include a wide variety of information types that would help them to function more effectively in their surrounding environments. Because of this, it would be very helpful if such persons could be supported by providing them with various kinds of event information in the form of visual data. The ultimate purpose of this research is to classify various types of environmental sounds that can provide important clues from the viewpoint of safe-living and then apply machine learning techniques to visually display that information in real time on smartglasses. Specifically, our method extracts the acoustic features of environmental sounds from their time-, frequency-, and time-frequency-domain characteristics and then classifies those sounds based on their extracted features. Ultimately, such information will be visually displayed on smartglasses, thus allowing hearing-impaired persons to process those environmental sounds while continuing to observe the world around them in a normal manner. In this paper, the validity of our empirical and academic research into machine learning-based environmental sound classifications is discussed.

MA-CNN: Multi-augmented data classification using 2D-CNN with kaiming initialization for environmental sound classification

MA-CNN: Multi-augmented data classification using 2D-CNN with kaiming initialization for environmental sound classification

Deep Convolutional Neural Networks For Environmental Sound Classification

Abstract: We propose a model to classify environmental sounds such as People Sounds, Vehicles Sounds, Siren Sounds, Horn, Engine Sounds. We perform Data Augmentation techniques to extract best features from the given audio to classify which class of sound. Our deep convolutional neural network architecture uses stacked convolutional and pooling layers to extract highlevel feature representations from spectrogram-like features from the given input.

Environmental Sound Classification Based on Transfer-Learning Techniques with Multiple Optimizers

The last decade has seen increased interest in environmental sound classification (ESC) due to the increased complexity and rich information of ambient sounds. The state-of-the-art methods for ESC are based on transfer learning paradigms that often utilize learned representations from common image-classification problems. This paper aims to determine the effectiveness of employing pre-trained convolutional neural networks (CNNs) for audio categorization and the feasibility of retraining. This study investigated various hyper-parameters and optimizers, such as optimal learning rate, epochs, and Adam, Adamax, and RMSprop optimizers for several pre-trained models, such as Inception, and VGG, ResNet, etc. Firstly, the raw sound signals were transferred into an image format (log-Mel spectrogram). Then, the selected pre-trained models were applied to the obtained spectrogram data. In addition, the effect of essential retraining factors on classification accuracy and processing time was investigated during CNN training. Various optimizers (such as Adam, Adamax, and RMSprop) and hyperparameters were utilized for evaluating the proposed method on the publicly accessible sound dataset UrbanSound8K. The proposed method achieves 97.25% and 95.5% accuracy on the provided dataset using the pre-trained DenseNet201 and the ResNet50V2 CNN models, respectively.

A new lateral geniculate nucleus pattern-based environmental sound classification using a new large sound dataset

Background and purposeOne of the essential purposes of sound classification is to achieve similar/over classification ability of the human auditory system (HAS). A new dataset and a biologically inspired feature extraction function have been proposed to realize this aim. We have developed a highly accurate sound classification architecture using the proposed biological-inspired feature extraction function. Materials and methodsIn this research, a new environmental sound classification (ESC) dataset has been collected as a testbed, and this dataset contains 5000 sounds with 50 classes. Moreover, the collected ESC sound dataset is balanced. A new hand-modeled sound classification model has been proposed to classify sounds of this dataset. This model consists of (i) feature generation using a new lateral geniculate nucleus pattern (LGNPat), statistical moments and discrete wavelet transform (DWT), (ii) loop-based (iterative) neighborhood component analysis (INCA) based feature selection, (iii) classification using the selected features by k nearest neighbors (kNN) with 10-fold cross-validation. The proposed sound classification architecture is an extendable model. In this aspect, a new generation of hand-modeled sound/one-dimensional signal classification methods can be proposed. ResultsThe presented hand-modeled learning method was applied to the ESC dataset acquired, and our LGNPat-based model attained 93.34% classification. ConclusionsThe computed 93.34% on the collected ESC dataset has demonstrated the success of our model. Moreover, the collected dataset has been publicly published. In this aspect, the published dataset can be used to improve advanced sound classification models.

A Deep Attention Model for Environmental Sound Classification from Multi-Feature Data

Automated environmental sound recognition has clear engineering benefits; it allows audio to be sorted, curated, and searched. Unlike music and language, environmental sound is loaded with noise and lacks the rhythm and melody of music or the semantic sequence of language, making it difficult to find common features representative enough of various environmental sound signals. To improve the accuracy of environmental sound recognition, this paper proposes a recognition method based on multi-feature parameters and time–frequency attention module. It begins with a pretreatment that relies on multi-feature parameters to extract the sound, which supplements the phase information lost by the Log-Mel spectrogram in the current mainstream methods, and enhances the expressive ability of input features. A time–frequency attention module with multiple convolutions is designed to extract the attention weight of the input feature spectrogram and reduce the interference coming from the background noise and irrelevant frequency bands in the audio. Comparative experiments were conducted on three general datasets: environmental sound classification datasets (ESC-10, ESC-50) and an UrbanSound8K dataset. Experiments demonstrated that the proposed method performs better.

Deep convolutional neural network for environmental sound classification via dilation

In the recent time, enviromental sound classification has received much popularity. This area of research comes under domain of non-speech audio classification. In this work, we have proposed a dilated Convolutional Neural Network approch to classify urban sound. We have carried out feature extraction, data augmentation techniques to carry out our experimental strategy smoothly. We also found out the activation maps of each layers of dilated convolution neural network. An increamental dilation rate has exploited Overall we achieved 84.16% of accuracy from the proposed dilated convolutional method. The gradual increaments of dilation rate has exploited the worse effect of grindding and has lowered down the computational cost. Also, overall classification performance, precision, recall,overall truth and kappa value have been obtained from our proposed method. We have considered 10 fold cross validation for the implementation of the dilated CNN model.

Research on Acoustic Events Recognition Method With Dimensionality Reduction Combining Attention and Mutual Information

The environment sound classification(ESC) is of great significance to the monitoring and control of urban noise. Aiming at the curse of dimensionality phenomenon in ESC, a feature dimensionality reduction architecture combining attention and mutual information is proposed. In order to match the two-dimensional MFCC (Mel Frequency Cepstral Coefficients) feature matrix, the proposed method separates and reconstructs the feature frames of different samples, and achieves the effect of dimensionality reduction by making decisions on the information entropy between the feature frames and labels. In addition, the method combines LSTM (Long Short-Term Memory) model with attention mechanism to ensure the recognition accuracy of the model after dimensionality reduction. Ten urban acoustic events from UrbanSound8k (US8K) dataset are selected to verify the performances of the proposed method by simulation experiments, which are also compared with the existing classification methods. The simulation results show that by combining the attention mechanism and mutual information, the recognition accuracy of the proposed method on the UrbanSound8k dataset is 95.16%, and the parameter scale is the smallest, only 0.92M. Moreover, the model parameter scale is adjustable by dynamic frame retention mechanism to balance the recognition accuracy and speed. This method not only ensures a high classification accuracy, but also can reduce computing power consumption and storage space of monitoring equipment, which shows a better practical performance for urban acoustic events recognition.

Fast environmental sound classification based on resource adaptive convolutional neural network

Recently, with the construction of smart city, the research on environmental sound classification (ESC) has attracted the attention of academia and industry. The development of convolutional neural network (CNN) makes the accuracy of ESC reach a higher level, but the accuracy improvement brought by CNN is often accompanied by the deepening of network layers, which leads to the rapid growth of parameters and floating-point operations (FLOPs). Therefore, it is difficult to transplant CNN model to embedded devices, and the classification speed is also difficult to accept. In order to reduce the hardware requirements of running CNN and improve the speed of ESC, this paper proposes a resource adaptive convolutional neural network (RACNN). RACNN uses a novel resource adaptive convolutional (RAC) module, which can generate the same number of feature maps as conventional convolution operations more cheaply, and extract the time and frequency features of audio efficiently. The RAC block based on the RAC module is designed to build the lightweight RACNN model, and the RAC module can also be used to upgrade the existing CNN model. Experiments based on public datasets show that RACNN achieves higher performance than the state-of-the-art methods with lower computational complexity.

Recognize the surrounding: Development and evaluation of convolutional deep networks using gammatone spectrograms and raw audio signals

Environmental sound (ES) consists of the surrounding sounds around us. The classification of this kind of sound has specific and significant contributions in many modern world applications, especially in the internet of things (IoT). However, accurately predicting the sound class of an ES is difficult because of the uncertain pattern in most cases. Although a lot of research has been done to develop a fully accurate environmental sound classification (ESC) system using different extracted features from signal, feature selection and classifier design is still a harsh job and sometimes does not guarantee a precise result. In recent years, ESC has attracted the attention of the research community because of the breakthrough in learning algorithms. Following this trend, a study is carried out here to find and propose two CNN models, 1D and 2D, for raw signal input and Gammatone spectrogram input, respectively. Compared with 2D CNN model, 1D CNN for time series waveform is a recent idea. Assessing the models on two various datasets (ESC-10 and US-8 K), the overall accuracy for 2D CNN is found 80.2% (ESC-10), 89% (US-8 K). For 1D CNN, the accuracy is 80.4% (ESC-10), 86% (US-8 K). The performance of the proposed 1D CNN model is almost similar to that of the proposed 2D CNN model. Effective data augmentation procedure is done to achieve this high accuracy by introducing variability in the given dataset, and to reduce overfitting. Finally, the proposed models have a minimal number of parameters to be trained, and floating-point operations per second (FLOPS) are also minimal, especially for 1D CNN.

Data augmentation guided knowledge distillation for environmental sound classification

Environmental sound classification (ESC) is an increasingly relevant field of research in recent years but has high computational overhead in its classification of environmental sounds. Knowledge distillation (KD) is a prominent technique to develop a lightweight deep model by distilling knowledge from a heavyweight model into a less computationally complex model. Generally, conventional KD techniques require manual setting of a temperature parameter to explore the similarity among the classes. Herein, we propose a novel data augmentation technique that creates an augmented data instance by blending hidden features of a data sample from one class with a style information (mean and standard values) of a data sample from another class. We have designed a new loss function to accomplish Knowledge Distillation that minimizes Kullback–Leibler (KL) divergence loss between class probabilities obtained from the teacher and student networks while classifying the augmented data sample. Furthermore, the proposed KD technique rids the process of a temperature parameter that needs to be set manually by the traditional vanilla KD technique. Our experiments on two benchmark ESC datasets i.e., the ESC-10 and DCASE 2019 Task-1(A) dataset demonstrate comparable performance of the student network to state-of-the-art techniques. Moreover, the student model is explainable and clearly explains why a signal is classified into a specific class.

Environmental Sound Classification via Time–Frequency Attention and Framewise Self-Attention-Based Deep Neural Networks

Environmental sound classification (ESC) is crucial to understanding the surroundings in Internet of Things (IoT) applications. The state-of-the-art deep learning approaches do not have good ESC performance when there exists various clutter interference, which is common in IoT scenarios. In this article, we present a novel deep neural network framework based on time–frequency attention and framewise self-attention (TFFS-DNN). It consists of two major novel architectures: 1) gradient and 2) latent feature-based DNN to generate our time–frequency attention, which can locate the relevant time–frequency (i.e., spectral) features accurately, and self-attention normalization DNN to generate our framewise self-attentions which properly indicate the relevance of frames. By conjoining these two sorts of distinct and complementary attentions with spectrograms, we are able to identify the importance or relevance in terms of time, frequency, and frame of the sounds using TFFS-DNN, which helps in distinguishing clutter such as background as well as model interpretation to some extent. Thus, the proposed TFFS-DNN can classify environmental sounds with clutter. The evaluation using four real-world environmental sound data sets demonstrates the superior performance of the proposed framework over several state-of-the-art schemes. Notably, we achieve 79.23% classification accuracy on the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">UrbanSound</i> data set, a raw environmental sound data set that is full of clutter. The ablation study demonstrates a relative 3%–9% improvement of classification accuracy by the proposed framework over the baseline deep model.

Comparative Analysis of Machine Learning Algorithms for Classification of Environmental Sounds and Fall Detection

In recent years, number of elderly people in population has been increased because of the rapid advancements in the medical field, which make it necessary to take care of old people. Accidental fall incidents are life-threatening and can lead to the death of a person if first aid is not given to the injured person. Immediate response and medical assistance are necessary in case of accidental fall incidents to elderly people. The research community explored various fall detection systems to early detect fall incidents, however, still there exist numerous limitations of the systems such as using expensive sensors, wearable sensors that are hard to wear all the time, camera violates the privacy of person, and computational complexity. In order to address the above-mentioned limitations of the existing systems, we proposed a novel set of integrated features that consist of melcepstral coefficients, gammatone cepstral coefficients, and spectral skewness. We employed a decision tree for the classification performance of both binary problems and multi-class problems. We obtained an accuracy of 91.39%, precision of 96.19%, recall of 91.81%, and F1-score of 93.95%. Moreover, we compared our method with existing state-of-the-art methods and the results of our method are higher than other methods. Experimental results demonstrate that our method is reliable for use in medical centers, nursing houses, old houses, and health care provisions.