Preserving Motion-Tolerant Contextual Visual Saliency for Video Resizing

Abstract

State of the art video resizing methods usually produce perceivable visual discontinuities, especially in videos containing significant motion. To resolve the problem, contextual information about the focus of interest in consecutive video frames should be considered in order to preserve the visual continuity. In this paper, to detect the focus of interest with motion-tolerance, we propose a novel approach for modelling visual dynamics based on spatiotemporal slices (STS), which provide rich visual patterns along a large temporal scale. First, patch-based visual patterns are computed to generate a codebook of the automatically specified spatiotemporal extent determined by the contextual information in the STS. The codebook is then used to compute its associated response in each video frame, and eventually an importance map covering the focus of interest in a video clip can be obtained. To preserve the visual continuity of the content, particularly an important area, a multi-cue approach is used to guide a mesh-based non-homogeneous warping operation constrained by the trajectories in the STS. For the performance evaluation, we present a novel measure that utilizes patch-based Kullback-Leibler divergence (KL-divergence) to gauge the deformation of the focus of interest under the proposed video resizing approach. Experimental results show that the STS-based approach can generate retargeted videos effectively, while maintaining their isotropic manipulation and the continuous dynamics of visual perception.