An audio–visual human attention analysis approach to abrupt change detection in videos

Abstract

Abrupt changes in videos, such as sudden running, usually indicate abnormal events and play a significant role in attracting human attention. We propose an approach to detect abrupt changes in videos based on Bayesian surprise theory, which considers both visual and audio modalities. Specifically, after generating surprise curves from the audio and visual modalities, we obtain a synchronized sequence based on the time-synchrony between audio–visual series in videos. The approach is fully automated and does not require any prior information. Experimental results from tests on human behavior and natural scene video datasets demonstrate that the proposed method is able to detect abrupt changes like sudden running or the collapse of an object. The proposed approach is further evaluated on the entire dataset we collected.