Visual Attention Prediction Research Articles

An extensive evaluation of deep featuresof convolutional neural networks for saliency prediction of human visual attention

Based on transfer learning, feature maps of deep convolutional neural networks (DCNNs) have been used to predict human visual attention. In this paper, we conduct extensive comparisons to investigate effects of feature maps on the predictions of the human visual attention using a deep features based saliency model framework. The feature maps of seven pretrained DCNNs are investigated using classical and class activation maps approaches. The performances of various saliency implementations are evaluated over four datasets using three metrics. The results demonstrate that deep feature maps of the pretrained DCNNs can be used to create saliency maps for the prediction of human visual attention. The incorporation of multiple levels of blurred and multi-scale feature maps improves the extraction of salient regions. Moreover, DCNNs pretrained using the Places dataset provide more localized objects that can be beneficial to the top-down saliency maps.

Bio-Inspired Representation Learning for Visual Attention Prediction.

Visual attention prediction (VAP) is a significant and imperative issue in the field of computer vision. Most of the existing VAP methods are based on deep learning. However, they do not fully take advantage of the low-level contrast features while generating the visual attention map. In this article, a novel VAP method is proposed to generate the visual attention map via bio-inspired representation learning. The bio-inspired representation learning combines both low-level contrast and high-level semantic features simultaneously, which are developed by the fact that the human eye is sensitive to the patches with high contrast and objects with high semantics. The proposed method is composed of three main steps: 1) feature extraction; 2) bio-inspired representation learning; and 3) visual attention map generation. First, the high-level semantic feature is extracted from the refined VGG16, while the low-level contrast feature is extracted by the proposed contrast feature extraction block in a deep network. Second, during bio-inspired representation learning, both the extracted low-level contrast and high-level semantic features are combined by the designed densely connected block, which is proposed to concatenate various features scale by scale. Finally, the weighted-fusion layer is exploited to generate the ultimate visual attention map based on the obtained representations after bio-inspired representation learning. Extensive experiments are performed to demonstrate the effectiveness of the proposed method.

Visual Saliency Prediction Based on Deep Learning

Human eye movement is one of the most important functions for understanding our surroundings. When a human eye processes a scene, it quickly focuses on dominant parts of the scene, commonly known as a visual saliency detection or visual attention prediction. Recently, neural networks have been used to predict visual saliency. This paper proposes a deep learning encoder-decoder architecture, based on a transfer learning technique, to predict visual saliency. In the proposed model, visual features are extracted through convolutional layers from raw images to predict visual saliency. In addition, the proposed model uses the VGG-16 network for semantic segmentation, which uses a pixel classification layer to predict the categorical label for every pixel in an input image. The proposed model is applied to several datasets, including TORONTO, MIT300, MIT1003, and DUT-OMRON, to illustrate its efficiency. The results of the proposed model are quantitatively and qualitatively compared to classic and state-of-the-art deep learning models. Using the proposed deep learning model, a global accuracy of up to 96.22% is achieved for the prediction of visual saliency.

Visual attention prediction for images with leading line structure

Researchers have proposed a wide variety of visual attention models, ranging from models that use local, low-level image features to recent approaches that incorporate semantic information. However, most models do not account for the visual attention evident in images with certain global structures. We focus specifically on “leading line” structures, in which explicit or implicit lines converge at a point. Through this study, we have conducted the experiments to investigate the visual attentions in images with leading line structure and propose new models that combine the low-level feature of center-surround differences of visual stimuli, the semantic feature of center bias and the structure feature of leading lines. We also create a new data set from 110 natural images containing leading lines and the eye-tracking data for 16 subjects. Our evaluation experiment showed that our models outperform the existing models against common indicators of saliency-map evaluation, underscoring the importance of leading lines in the modeling of visual attention.

Hybrid convolutional neural networks and optical flow for video visual attention prediction

In this paper, a convolutional neural networks (CNN) and optical flow based method is proposed for prediction of visual attention in the videos. First, a deep-learning framework is employed to extract spatial features in frames to replace those commonly used handcrafted features. The optical flow is calculated to obtain the temporal feature of the moving objects in video frames, which always draw audiences’ attentions. By integrating these two groups of features, a hybrid spatial temporal feature set is obtained and taken as the input of a support vector machine (SVM) to predict the degree of visual attention. Finally, two publicly available video datasets were used to test the performance of the proposed model, where the results have demonstrated the efficacy of the proposed approach.

Prediction of visual attention with deep CNN on artificially degraded videos for studies of attention of patients with Dementia

Studies of visual attention of patients with Dementia such as Parkinson’s Disease Dementia and Alzheimer Disease is a promising way for non-invasive diagnostics. Past research showed, that people suffering from dementia are not reactive with regard to degradations on still images. Attempts are being made to study their visual attention relatively to the video content. Here the delays in their reactions on novelty and “unusual” novelty of the visual scene are expected. Nevertheless, large-scale screening of population is possible only if sufficiently robust automatic prediction models can be built. In the medical protocols the detection of Dementia behavior in visual content observation is always performed in comparison with healthy, “normal control” subjects. Hence, it is a research question per see as to develop an automatic prediction models for specific visual content to use in psycho-visual experience involving Patients with Dementia (PwD). The difficulty of such a prediction resides in a very small amount of training data. In this paper the reaction of healthy normal control subjects on degraded areas in videos was studied. Furthermore, in order to build an automatic prediction model for salient areas in intentionally degraded videos for PwD studies, a deep learning architecture was designed. Optimal transfer learning strategy for training the model in case of very small amount of training data was deployed. The comparison with gaze fixation maps and classical visual attention prediction models was performed. Results are interesting regarding the reaction of normal control subjects against degraded areas in videos.

Learning to Model Task-Oriented Attention

For many applications in graphics, design, and human computer interaction, it is essential to understand where humans look in a scene with a particular task. Models of saliency can be used to predict fixation locations, but a large body of previous saliency models focused on free-viewing task. They are based on bottom-up computation that does not consider task-oriented image semantics and often does not match actual eye movements. To address this problem, we collected eye tracking data of 11 subjects when they performed some particular search task in 1307 images and annotation data of 2,511 segmented objects with fine contours and 8 semantic attributes. Using this database as training and testing examples, we learn a model of saliency based on bottom-up image features and target position feature. Experimental results demonstrate the importance of the target information in the prediction of task-oriented visual attention.

Predicting human gaze beyond pixels

A large body of previous models to predict where people look in natural scenes focused on pixel-level image attributes. To bridge the semantic gap between the predictive power of computational saliency models and human behavior, we propose a new saliency architecture that incorporates information at three layers: pixel-level image attributes, object-level attributes, and semantic-level attributes. Object- and semantic-level information is frequently ignored, or only a few sample object categories are discussed where scaling to a large number of object categories is not feasible nor neurally plausible. To address this problem, this work constructs a principled vocabulary of basic attributes to describe object- and semantic-level information thus not restricting to a limited number of object categories. We build a new dataset of 700 images with eye-tracking data of 15 viewers and annotation data of 5,551 segmented objects with fine contours and 12 semantic attributes (publicly available with the paper). Experimental results demonstrate the importance of the object- and semantic-level information in the prediction of visual attention.

An edge detection with automatic scale selection approach to improve coherent visual attention model

An automatic scale selection approach is developed to improve the coherent visual attention model (Le Meur, O., Le Callet, P., Barba, D., Thoreau, D., 2006. A coherent computational approach to model bottom-up visual attention. IEEE Trans. Pattern Anal. Machine Intell. 28 (5), 802–817). The new approach uses linear regression to combine the automatic scale selection attention model with the coherent visual attention model. It is biologically more plausible because two important properties (i.e. edge detection and scale selection) of human vision are taken into account. Its performance is evaluated using a large human fixation dataset. The t-test indicates that the improved model outperforms the coherent visual attention model highly significantly in both the non-weighting and weighting cases. The new model also outperforms seven other state-of-the-art saliency prediction models highly significantly (p<0.01). Thus it furnishes a more accurate model for human visual attention prediction.

Visual Attention Prediction Using Saliency Determination of Scene Understanding for Social Robots

In this paper, the biological ability of visual attention is modeled for social robots to understand scenes and circumstance. Visual attention is determined by evaluating visual stimuli and prior knowledge in the intelligent saliency searching. Visual stimuli are measured using information entropy and biological color sensitivities, where the information entropy evaluates information qualities and the color sensitivity assesses biological attraction of a presented scene. We also learn and utilize the prior knowledge of people’s focus in the prediction of visual attention. The performance of the proposed technique is studied on different sorts of natural scenes and evaluated with fixation data of actual eye-tracking database. The experimental results proved the effectiveness of the proposed technique in discovering salient regions and predicting visual attention. The robustness of the proposed technique to transformation and illumination variance is also investigated. Social robots equipped with the proposed technique can autonomously determine their attention to a scene autonomously so as to behave naturally in the human robot interaction.

Learning-Based Prediction of Visual Attention for Video Signals

Visual attention, which is an important characteristic of human visual system, is a useful clue for image processing and compression applications in the real world. This paper proposes a computational scheme that adopts both low-level and high-level features to predict visual attention from video signal by machine learning. The adoption of low-level features (color, orientation, and motion) is based on the study of visual cells, and the adoption of the human face as a high-level feature is based on the study of media communications. We show that such a scheme is more robust than those using purely single low- or high-level features. Unlike conventional techniques, our scheme is able to learn the relationship between features and visual attention to avoid perceptual mismatch between the estimated salience and the actual human fixation. We also show that selecting the representative training samples according to the fixation distribution improves the efficacy of regressive training. Experimental results are shown to demonstrate the advantages of the proposed scheme.