A novel audio-visual based beamformer using a 360° camera and planar microphone array system

Abstract

Beamforming systems that rely only on microphone arrays may suffer from poor performance in complex acoustic environments with multiple sources and room reverberation. To overcome this limitation, we propose a novel method that combines a planar microphone array and a 360° camera to steer the beamformer. The camera provides visual information about the location and orientation of sound sources, which can be used to adjust the beamformer parameters. We will present results that compare the signal-to-noise ratio (SNR) of the proposed method with a conventional beamforming microphone-based method in a simulated room with different source positions, orientations, and noise levels to investigate the benefit of visual information for beamforming in different environments. The proposed method has the potential to improve the performance of beamforming systems in applications such as speech enhancement, speech recognition, and speaker localization.