Deep saliency models learn low-, mid-, and high-level features to predict scene attention

Taylor R Hayes,John M Henderson

doi:10.1038/s41598-021-97879-z

Abstract

Deep saliency models represent the current state-of-the-art for predicting where humans look in real-world scenes. However, for deep saliency models to inform cognitive theories of attention, we need to know how deep saliency models prioritize different scene features to predict where people look. Here we open the black box of three prominent deep saliency models (MSI-Net, DeepGaze II, and SAM-ResNet) using an approach that models the association between attention, deep saliency model output, and low-, mid-, and high-level scene features. Specifically, we measured the association between each deep saliency model and low-level image saliency, mid-level contour symmetry and junctions, and high-level meaning by applying a mixed effects modeling approach to a large eye movement dataset. We found that all three deep saliency models were most strongly associated with high-level and low-level features, but exhibited qualitatively different feature weightings and interaction patterns. These findings suggest that prominent deep saliency models are primarily learning image features associated with high-level scene meaning and low-level image saliency and highlight the importance of moving beyond simply benchmarking performance.

Highlights

Deep saliency models represent the current state-of-the-art for predicting where humans look in real-world scenes
We examined each deep saliency model separately by fitting a separate logistic general linear mixed effects (GLME) model for each deep saliency model
Within each GLME model, whether a region was fixated (1) or not (0) was the dependent variable and the scene region’s mean deep saliency model value (MSI-Net Fig. 1b; DeepGaze[2], Fig. 1c; SAM-ResNet Fig. 1d), mean center proximity value (Fig. 1e), and the deep saliency by center proximity interaction were treated as predictors

Summary

Introduction

Deep saliency models represent the current state-of-the-art for predicting where humans look in real-world scenes. For deep saliency models to inform cognitive theories of attention, we need to know how deep saliency models prioritize different scene features to predict where people look. Theories of attention focused on the role of low-level feature differences in capturing attention and were based on experiments using simple stimuli like lines and/or basic shapes that varied in low-level features like orientation, color, luminance, texture, shape, or motion[10,11,12] These early theories were formalized into computational image ‘saliency’ models that combined the different low-level feature maps based on mechanisms observed in early visual cortex such as center-surround dynamics to generate quantitative predictions in the form of ‘saliency maps’[4,5,13,14,15]. Biological, and computational work on the role of low-level features in guiding attention, it will be important to quantify the degree to which low-level features are associated with deep saliency model performance

Methods

Results

Conclusion