Temporally Consistent Sound Source Localization via Self-Supervised Losses

Abstract

Humans derive contextual information from auditory and visual cues. In this thesis, we focus on improving sound source localization in videos, which is the correlation of au- dio signals to pixels. In this thesis, we study self-supervised losses with the goal of improving both accuracy and temporal consistency. We introduce a new evaluation benchmark that includes temporal ground truth segmentations, we refer to this benchmark as the "Temporally Way Harder" dataset. We demonstrate both qualitatively and quantitatively that our method outperforms current state-of-the-art sound source localization methods in terms of accuracy and temporal consistency on an extant data set and our introduced data set. We motivate our work with ablations showing the existence of centre-bias and temporal inconsistency in past work. Furthermore, we introduce a quantitative metric for examining temporal consistency in sound source localization.