Divide and Distill: New Outlooks on Knowledge Distillation for Environmental Sound Classification

Abstract

Environmental sound classification (ESC) is an important research problem with a broad range of applications including audio-based surveillance, audio-visual systems, smart homes, and robotics, among others. The recently proposed vision multi-layer perceptron-mixer (MLP-mixer) has outperformed traditional deep models (CNN or ResNet) and attained new state-of-the-art performances for several computer vision applications (image/video classification and image segmentation). Following the success of MLP-mixer, in this paper, we propose a novel audio MLP-mixer (AMM) network that classifies the different types of environmental sounds. Despite the higher performance, the high computational cost (number of trainable parameters and floating point operations) prohibits deployment of the AMM model on edge for designing real-life applications. To alleviate the aforementioned issue, in this work, we present three different knowledge distillation (KD) strategies to train a compact deep network for ESC. The proposed strategies divide the input Mel-spectrogram into patches and a lightweight deep ESC model is trained in the presence of three teacher networks under the offline KD training framework. Additionally, we have designed two novel loss functions for KD that are free from a temperature parameter that need to be set manually by a user as in the case of the traditional vanilla KD technique. We conducted our experiments on three benchmark ESC datasets namely ESC-10, Urbansound8k (US8K), and DCASE-2019 Task-1(A). The obtained results demonstrate the significance of utilization of proposed methods over other existing KD methods in terms of classification accuracy.