Influence of visual information on the perception of environmental sounds

Abstract

The purpose of this study is to clarify the influence of visual information on perception of environmental sounds. Audiovisual stimuli (360-degree video and ambisonic recordings) sampled at 6 locations in a region were presented to subjects through VR system with headphone and were rated by subjects in various evaluation items on a 7-step categorical scale. Stimuli presentation has three modes: only auditory stimulation (auditory single-mode), only visual stimulation (visual single-mode), and simultaneous presentation of audiovisual stimulation (multi-mode). Evaluation results were subjected to analysis of variance, significance tests and factor analysis to clarify differences in evaluation tendencies between modes. The dispersion of the evaluation values for auditory aspect under multi-mode tends to be smaller than that under auditory single-mode. It was found that visual information generally has the influence of relieving the impression of prominent environmental sounds. In addition, multi-mode evaluation tends to be related with evaluation obtained in visual single-mode, especially in evaluation items common to audio and visual. And it will be tried to explain the evaluation results in relation to the expected sound state for visual stimuli.