Sound-based multiple-equipment activity recognition using convolutional neural networks

Abstract

Automatically recognizing activities of heavy construction equipment using sound data has recently received considerable attention as a promising research area in construction. Although existing methods are effective, they only focus on tracking the activities of one single piece of equipment. On construction job sites, multiple equipment sound signals are mixed in the environment; Thus, there is a need for a robust method to recognize these activities that are taking place simultaneously. To address this shortcoming, we proposed a multi-label multi-level sound classification method based on Short-Time Fourier Transform (STFT) and Convolutional Neural Network (CNN) that only requires a single-channel off-the-shelf microphone. In addition, we developed a data augmentation method to simulate real-world equipment sound mixtures. We tested the proposed method on both synthetic and real-world equipment sound mixtures. The results of our study showed that this method was effective in identifying activities of multiple pieces of equipment on real construction job sites without the need for separating sound signals in advance. Future studies can focus on other potential applications of sound signal processing in the construction domain, including analyzing engine abnormalities and monitoring environmental performance of the equipment.