Acoustic Scene Classification Research Articles

An event-scene cooperative analysis network with dual-stream attention convolution module and soft parameter-sharing

Sound event detection (SED) and acoustic scene classification (ASC) are critical tasks in computational environmental sound scene analysis. Both exhibit significant potential across diverse applications, including urban environment detection, smart home integration, and anomaly monitoring. Considering the intricate relationship between acoustic scenes and sound events, researchers have proposed a joint analysis method for SED and ASC tasks using multi-task learning (MTL). However, conventional MTL-based methods rely mainly on a hard parameter-sharing mechanism, which restricts information flow and collaborative learning among tasks during training. To address this issue, this paper proposes a novel MTL method for SED that employs a soft parameter-sharing mechanism to explore the intrinsic correlation between acoustic scenes and sound events. A cross-stitch mechanism is introduced in this study to facilitate soft parameter-sharing between the SED and ASC tasks by employing scene information to improve the performance of the SED tasks and ensure training efficiency. Furthermore, a dual-stream attention convolution module (DACM) is designed to capture global and local contextual information through parallel branches of attention and convolution. To evaluate the performance of the proposed method in SED tasks, we tested two datasets: the TUT Acoustic Scenes 2016/2017 and TUT Sound Events 2016/2017 datasets and the synthetic sound scene datasets. Experimental results indicate that the proposed MTL method outperforms state-of-the-art approaches, demonstrating that acoustic scene information can improve the performance of SED tasks.

Deep semantic learning for acoustic scene classification

Acoustic scene classification (ASC) is the process of identifying the acoustic environment or scene from which an audio signal is recorded. In this work, we propose an encoder-decoder-based approach to ASC, which is borrowed from the SegNet in image semantic segmentation tasks. We also propose a novel feature normalization method named Mixup Normalization, which combines channel-wise instance normalization and the Mixup method to learn useful information for scene and discard specific information related to different devices. In addition, we propose an event extraction block, which can extract the accurate semantic segmentation region from the segmentation network, to imitate the effect of image segmentation on audio features. With four data augmentation techniques, our best single system achieved an average accuracy of 71.26% on different devices in the Detection and Classification of Acoustic Scenes and Events (DCASE) 2020 ASC Task 1A dataset. The result indicates a minimum margin of 17% against the DCASE 2020 challenge Task 1A baseline system. It has lower complexity and higher performance compared with other state-of-the-art CNN models, without using any supplementary data other than the official challenge dataset.

Acoustic Scene Classification Across Cities and Devices via Feature Disentanglement

Acoustic Scene Classification Across Cities and Devices via Feature Disentanglement

An Attention-Based Time-Frequency Pyramid Pooling Strategy in Deep Convolutional Networks for Acoustic Scene Classification

An Attention-Based Time-Frequency Pyramid Pooling Strategy in Deep Convolutional Networks for Acoustic Scene Classification

Cooperative Scene-Event Modelling for Acoustic Scene Classification

Cooperative Scene-Event Modelling for Acoustic Scene Classification

A CNN-Based Multi-Scale Pooling Strategy for Acoustic Scene Classification

A CNN-Based Multi-Scale Pooling Strategy for Acoustic Scene Classification

Instance-level loss based multiple-instance learning framework for acoustic scene classification

An acoustic scene is inferred by detecting properties combining diverse sounds and acoustic environments. This study is intended to discover these properties effectively using multiple-instance learning (MIL). MIL, also known as a weakly supervised learning approach, is a strategy for extracting an instance vector from an audio chunk that composes an audio clip and utilizing these unlabeled instances to infer a scene corresponding to the input data. However, many studies pointed out an underestimation problem of MIL. In this study, we propose an enhanced MIL framework more suitable for ASC systems by defining instance-level labels and loss to extract and cluster instances effectively. Furthermore, we design a lightweight convolutional neural network named FUSE comprising frequency-, temporal-sided depthwise, and pointwise convolutional filters. Experimental results show that the confidence and proportion of positive instances significantly increase compared to vanilla MIL, overcoming the underestimation problem and improving the classification accuracy even higher than the supervised learning. The proposed system achieved a performance of 81.1%, 72.3%, and 58.3% on the TAU urban acoustic scenes 2019, 2020 mobile, and 2022 mobile datasets with 139 K parameters, respectively. In particular, it achieves the highest performance among the systems having under the 1 M parameters for the TAU urban acoustic scenes 2019 dataset.

Predictions for sound events and soundscape impressions from environmental sound using deep neural networks

In this study, we investigate methods for quantifying soundscape impressions, meaning pleasantness and eventfulness, from environmental sounds. From a point of view of Machine Learning (ML) research areas, acoustic scene classification (ASC) tasks and sound event classification (SEC) tasks are intensively studied and their results are helpful to consider soundscape impressions. In general, while most of ASCs and SECs use only sound for the classifications, a soundscape impression should not be perceived just from a sound but perceived from a sound with a landscape by human beings. Therefore, to establish automatic quantification of soundscape impressions, it should use other information such as landscape in addition to sound. First, we tackle to predict of two soundscape impressions using sound data collected by the cloud sensing method. For this purpose, we have proposed prediction method of soundscape impressions using environmental sounds and aerial photographs. Second, we also tackle to establish environmental sound classification using feature extractor trained by Variational Autoencoder (VAE). The feature extractor by VAE can be trained as unsupervised learning, therefore, it could be promising approaches for the growth dataset like as our cloud-sensing data collection schemes. Finally, we discuss about an integration for these methods.

Sound Event Detection with Perturbed Residual Recurrent Neural Network

Sound event detection (SED) is of great practical and research significance owing to its wide range of applications. However, due to the heavy reliance on dataset size for task performance, there is often a severe lack of data in real-world scenarios. In this study, an improved mean teacher model is utilized to carry out semi-supervised SED, and a perturbed residual recurrent neural network (P-RRNN) is proposed as the SED network. The residual structure is employed to alleviate the problem of network degradation, and pre-training the improved model on the ImageNet dataset enables it to learn information that is beneficial for event detection, thus improving the performance of SED. In the post-processing stage, a customized median filter group with a specific window length is designed to effectively smooth each type of event and minimize the impact of background noise on detection accuracy. Experimental results conducted on the publicly available Detection and Classification of Acoustic Scenes and Events 2019 Task 4 dataset demonstrate that the P-RRNN used for SED in this study can effectively enhance the detection capability of the model. The detection system achieves a Macro Event-based F1 score of 38.8% on the validation set and 40.5% on the evaluation set, indicating that the proposed method can adapt to complex and dynamic SED scenarios.

Hierarchical classification for acoustic scenes using deep learning

Acoustic Scene Classification (ASC) aims to obtain the sound environment by analyzing audio signals. Due to the low complexity and acquisition cost of audio signals, ASC has enormous potential in various applications, such as audio-based surveillance, smart cities/homes, and robotics. Recently, various methods have been proposed for ASC and achieved good performance. However, when they are used to address complex ASC problems, most of them suffer from the low-performance problem. In this paper, we propose to use hierarchical classification methods to replace the conventional flat approach in ASC applications, which utilizes the class hierarchy to optimize classification performance. In particular, we investigate the ASC problem under the framework of hierarchical classification. Firstly, to improve classification performance, three hierarchical classification methods introducing the class hierarchy of acoustic scenes are proposed for ASC. Moreover, to fully utilize the class hierarchy, a hybrid hierarchical classification method, and an optimal late fusion-based hierarchical method are proposed, which are based on the flexibility and simplification of hierarchical classification. The experiments demonstrate the efficacy of hierarchical ASC systems for performance improvement, and the best system achieves an accuracy of 78.86% on the DCASE 2020 Task1A dataset, resulting in accuracy gains of 24.76% and 8.52% absolute over the DCASE 2020 Task 1A baseline and the conventional non-hierarchical method, respectively.

A Squeeze-and-Excitation and Transformer-Based Cross-Task Model for Environmental Sound Recognition

Environmental sound recognition (ESR) is an emerging research topic in audio pattern recognition. Many tasks are presented to resort to computational systems for ESR in real-life applications. However, current systems are usually designed for individual tasks, and are not robust and applicable to other tasks. Cross-task systems, which promote unified knowledge modeling across various tasks, have not been thoroughly investigated. In this paper, we propose a cross-task system for three different tasks of ESR: acoustic scene classification, urban sound tagging, and anomalous sound detection. An architecture named SE-Trans is presented that uses attention mechanism-based Squeeze-and-Excitation and Transformer encoder modules to learn channel-wise relationship and temporal dependencies of the acoustic features. FMix is employed as the data augmentation method that improves the performance of ESR. Evaluations for the three tasks are conducted on the recent databases of DCASE challenges. The experimental results show that the proposed cross-task system achieves state-of-the-art performance on all tasks. Further analysis demonstrates that the proposed cross-task system can effectively utilize acoustic knowledge across different ESR tasks.

Bi-level Acoustic Scene Classification Using Lightweight Deep Learning Model

Bi-level Acoustic Scene Classification Using Lightweight Deep Learning Model

Online Continual Learning in Acoustic Scene Classification: An Empirical Study.

Numerous deep learning methods for acoustic scene classification (ASC) have been proposed to improve the classification accuracy of sound events. However, only a few studies have focused on continual learning (CL) wherein a model continually learns to solve issues with task changes. Therefore, in this study, we systematically analyzed the performance of ten recent CL methods to provide guidelines regarding their performances. The CL methods included two regularization-based methods and eight replay-based methods. First, we defined realistic and difficult scenarios such as online class-incremental (OCI) and online domain-incremental (ODI) cases for three public sound datasets. Then, we systematically analyzed the performance of each CL method in terms of average accuracy, average forgetting, and training time. In OCI scenarios, iCaRL and SCR showed the best performance for small buffer sizes, and GDumb showed the best performance for large buffer sizes. In ODI scenarios, SCR adopting supervised contrastive learning consistently outperformed the other methods, regardless of the memory buffer size. Most replay-based methods have an almost constant training time, regardless of the memory buffer size, and their performance increases with an increase in the memory buffer size. Based on these results, we must first consider GDumb/SCR for the continual learning methods for ASC.

An Integrated Convolutional Neural Network with a Fusion Attention Mechanism for Acoustic Scene Classification

An Integrated Convolutional Neural Network with a Fusion Attention Mechanism for Acoustic Scene Classification

Lightweight deep neural networks for acoustic scene classification and an effective visualization for presenting sound scene contexts

In this paper, we propose lightweight deep neural networks for Acoustic Scene Classification (ASC) and a visualization method for presenting a sound scene context. To this end, we first propose an inception-based and low-memory footprint ASC model as the ASC baseline. The ASC baseline is then compared with benchmark and high-complexity network architectures. Next, we improve the ASC baseline by proposing a novel deep neural network architecture which leverages a residual-inception architecture and multiple kernels. Given the novel residual-inception (NRI) based model, we apply multiple techniques of model compression to evaluate the trade off between the model complexity and the model accuracy performance. Finally, we evaluate whether sound events detected in a sound scene recording can help to improve ASC accuracy performance and to present the sound scene context more comprehensively. We conduct extensive experiments on various ASC datasets, including sound scene datasets proposed for IEEE AASP Challenge on Detection and Classification of Acoustic Scenes and Events (DCASE) 2018 Task 1A and 1B, 2019 Task 1A and 1B, 2020 Task 1A, 2021 Task 1A, and 2022 Task 1. Our experimental results on several different ASC challenges highlight two main achievements. First, given the analysis of the trade off between the model performance and the model complexity, we propose two low-complexity ASC models: The medium-size model (MM) presents 4.96 M trainable parameters, 19.3 MB memory occupation, and 7.12 BFLOPs; The small-size model (SM) presents a very low complexity of 120 K trainable parameters, 120 KB memory occupation, and 0.82 BFLOPs. These ASC systems are very competitive to the state-of-the-art systems and compatible for real-life applications on a wide range of edge devices. Secondly, from the analysis of the role of sound events in a sound scene, we propose an effective visualization method for comprehensively presenting a sound scene context. By combining both the sound scene and sound event information, the visualization method not only indicates predicted sound scene contexts with high probabilities but also provides statistics of sound events occurring in these sound scene contexts.

Multi-level distance embedding learning for robust acoustic scene classification with unseen devices

Multi-level distance embedding learning for robust acoustic scene classification with unseen devices

Frequency-based CNN and attention module for acoustic scene classification

Acoustic scene classification (ASC) is an audio classification task that identifies the environment in which sounds are recorded. Audio-related machine learning algorithms suffer from the device mismatch problem; that is, when trained from audio data recorded from one device, the algorithms cannot generalize to audio samples recorded using another device. In this study, a novel convolutional neural network, called a frequency-aware convolutional neural network (FACNN), is introduced to solve the device mismatch problem by focusing on the frequency information of the audio samples. Furthermore, an attention module, called the frequency attention network (FANet), is introduced to generate an attention map based on the frequency information of the input feature maps. FANet helps the FACNN to focus on the important frequency information, thus improving performance. The proposed method is trained on the TAU Urban Acoustic Scenes 2019 Mobile development dataset and TAU Urban Acoustic Scenes 2020 Mobile development dataset. The proposed method achieves a state-of-the-art accuracy of 75.99% in the TAU Urban Acoustic Scenes 2019 Mobile development dataset and a competitive result of 72.6% in the TAU Urban Acoustic Scenes 2020 Mobile development dataset. In addition, a comparison of FANet with the convolutional block attention module (CBAM) and the squeeze-and-excitation network (SENet) was performed. The results show that FANet can mitigate the device mismatch problem by improving the performance of the unseen devices.

Acoustic Scene Classification using Deep Fisher network

Acoustic Scene Classification (ASC) is the task of assigning a semantic label to an audio recording, based on the surrounding environment. In this work, a Fisher network is introduced for ASC. The proposed method mimics the working mechanism of a feed-forward Convolutional Neural Network (CNN) where, output of a layer is fed as an input to the succeeding layer. The Fisher network consists of a feature extraction step followed by a Fisher layer. The Fisher layer has three sub-layers, namely, Fisher Vector (FV) encoder, temporal pyramid and normalization layers along with feature reduction layer. Gammatone Time Cepstral Coefficients (GTCCs) and Mel-spectrograms are the features encoded as Fisher vector representation in FV encoder sub-layer. Temporal information of the Fisher vectors is retained using temporal pyramid sub-layer. After temporal pyramids are extracted from Fisher vectors, they are available as a feature vector. Irrelevant dimensions of the temporal pyramids are reduced further using Principal Component Analysis (PCA) in normalization and PCA sub-layers. The proposed model is evaluated on five DCASE datasets, TUT Urban Acoustic Scenes 2018 and Mobile, DCASE 2019 Acoustic Scene Classification Task 1(a) and Task 1(b), TAU Urban Acoustic Scenes 2020 datasets. The overall classification accuracy is 93%, 91%, 92%, 91% and 89% for TUT 2018, TUT Mobile 2018, DCASE Task 1(a) 2019, DCASE Task 1(b) 2019, and TAU Urban Acoustic Scenes 2020 datasets, respectively. The proposed model performed much better than the state-of-the-art ASC systems.

A Genetic Algorithm Approach to Automate Architecture Design for Acoustic Scene Classification

Convolutional neural networks (CNNs) have been widely used with remarkable success in the acoustic scene classification (ASC) task. However, the performance of these CNNs highly relies on their architectures, requiring a lot of effort and expertise to design CNNs suitable for the investigated problem. In this work, we propose an efficient genetic algorithm (GA) that aims to find optimized CNN architectures for the ASC task. The proposed algorithm uses frequency-dimension splitting of the input spectrograms in order to explore the architecture search space in sub-CNN models in addition to classical single-path CNNs. Specifically, this algorithm aims to find the best number of sub-CNNs in addition to their architectures to better capture the distinct features of the input spectrograms. The proposed GA is specifically designed for sound classification to suit the ASC task than many other GAs that optimize conventional single-path CNN architectures. Experimental results on three benchmark datasets demonstrate the effectiveness of the proposed method. Specifically, the proposed algorithm has achieved around 17.8%, 16%, and 17.2%, relative improvement in accuracy with respect to the baseline systems on the development datasets of DCASE2018-Task1A, DCASE2019-Task1A, and DCASE2020-Task1A, respectively.

Sound Event Localization and Detection Using Imbalanced Real and Synthetic Data via Multi-Generator

This study proposes a sound event localization and detection (SELD) method using imbalanced real and synthetic data via a multi-generator. The proposed method is based on a residual convolutional neural network (RCNN) and a transformer encoder for real spatial sound scenes. SELD aims to classify the sound event, detect the onset and offset of the classified event, and estimate the direction of the sound event. In Detection and Classification of Acoustic Scenes and Events (DCASE) 2022 Task 3, SELD is performed with a few real spatial sound scene data and a relatively large number of synthetic data. When a model is trained using imbalanced data, it can proceed by focusing only on a larger number of data. Thus, a multi-generator that samples real and synthetic data at a specific rate in one batch is proposed to prevent this problem. We applied the data augmentation technique SpecAugment and used time-frequency masking to the dataset. Furthermore, we propose a neural network architecture to apply the RCNN and transformer encoder. Several models were trained with various structures and hyperparameters, and several ensemble models were obtained by "cherry-picking" specific models. Based on the experiment, the single model of the proposed method and the model applied with the ensemble exhibited improved performance compared with the baseline model.