GBVS360, BMS360, ProSal: Extending existing saliency prediction models from 2D to omnidirectional images

Abstract

In this paper it is studied how existing visual saliency models designed for 2D images can be extended and applied to Omnidirectional images in the equirectangular format. Three different frameworks, BMS360, GBVS360 and Projected Saliency were designed to address this task. In the particular case of BMS360 and GBVS360, the 2D models Boolean Map Saliency (BMS) and Graph-Based Visual Saliency (GBVS) were rigorously extended to integrate the specific properties of equirectangular images to model visual attention in omnidirectional images watched using Head Mounted Displays (HMDs). With the proposed extensions, the saliency prediction performance in comparison to their original design could significantly be improved. Another key result of this paper comes from the design of a framework called projected saliency which allows applying existing saliency models on equirectangular images without requiring in-depth adaptation of the prediction algorithms. The projected saliency framework enabled to study the interaction of features in the feature activation process of computational saliency models. It shows that the activation should not be done per-viewport, but should account for neighboring regions. As a consequence, we argue that the feature activation process should be performed at a global level instead of a local one. Here, the use of a local approach with a large FOV was found to be a good compromise between global activation and appropriate feature computation to directly port existing saliency models designed for rectilinear images to the domain of equirectangular images. In this paper an adaptive equatorial prior is also described. This equatorial prior allows to account for pitch shift of the horizon line. Finally, results show that the newly designed BMS360 and GVBS360 outperform even recent state-of-the-art models at head motion prediction and are competitive at head/eye saliency-map prediction. Moreover, all models and their code are made publicly available on GitHub allowing to further iterate on this work.