Visual Attention Model Research Articles

Saliency Priority of Individual Bottom-Up Attributes in Designing Visual Attention Models

A key factor in designing saliency detection algorithms for videos is to understand how different visual cues affect the human perceptual and visual system. To this end, this article investigated the bottom-up features including color, texture, and motion in video sequences for a one-by-one scenario to provide a ranking system stating the most dominant circumstances for each feature. In this work, it is considered the individual features and various visual saliency attributes investigated under conditions in which the authors had no cognitive bias. Human cognition refers to a systematic pattern of perceptual and rational judgments and decision-making actions. First, this paper modeled the test data as 2D videos in a virtual environment to avoid any cognitive bias. Then, this paper performed an experiment using human subjects to determine which colors, textures, motion directions, and motion speeds attract human attention more. The proposed benchmark ranking system of salient visual attention stimuli was achieved using an eye tracking procedure.

A robust model for salient text detection in natural scene images using MSER feature detector and Grabcut

Visual attention models have been used to recognize the most prominent region in a natural scene. These regions are going to pull the human attention. The state-of-art models keep on under-predicting the significant image regions having text. These are specifically the regions with most noteworthy semantic significance in a natural scene and turn out to be useful for saliency-based applications like image classification and captioning. The text or character detection as a salient region in image remains a challenging research problem. Text contents within the scene convey vital information about the image. For example, signboard content conveys the important information for visually impaired person. In this paper, we have proposed a new model for salient text detection in a natural scene. In the proposed model, we integrate saliency model with the segmentation and text detection approach in a natural scene to generate the text saliency. The experimental outcomes in ROC curve and DET curves illustrate that the proposed model outperformed the state-of-art methods for detection of salient text content from a natural scene.

Traffic signal detection and classification in street views using an attention model

Detecting small objects is a challenging task. We focus on a special case: the detection and classification of traffic signals in street views. We present a novel framework that utilizes a visual attention model to make detection more efficient, without loss of accuracy, and which generalizes. The attention model is designed to generate a small set of candidate regions at a suitable scale so that small targets can be better located and classified. In order to evaluate our method in the context of traffic signal detection, we have built a traffic light benchmark with over 15,000 traffic light instances, based on Tencent street view panoramas. We have tested our method both on the dataset we have built and the Tsinghua–Tencent 100K (TT100K) traffic sign benchmark. Experiments show that our method has superior detection performance and is quicker than the general faster RCNN object detection framework on both datasets. It is competitive with state-of-the-art specialist traffic sign detectors on TT100K, but is an order of magnitude faster. To show generality, we tested it on the LISA dataset without tuning, and obtained an average precision in excess of 90%.

A novel superpixel-based saliency detection model for 360-degree images

Effective visual attention modeling is a key factor that helps enhance the overall Quality of Experience (QoE) of VR/AR data. Although a huge number of algorithms have been developed in recent years to detect salient regions in flat-2D images, the research on 360-degree image saliency is limited. In this study, we propose a superpixel-level saliency detection model for 360-degree images by figure-ground law of Gestalt theory. First, the input image is segmented into superpixels. CIE Lab color space is then used to extract the perceptual features. We extract luminance and texture features for 360-degree images from L channel, while color features are extracted from a and b channels. We compute two components for saliency prediction by figure-ground law of Gestalt theory: feature contrast and boundary connectivity. The feature contrast is computed on superpixel level by luminance and color features. The boundary connectivity is predicted for background measure and it describes the spatial layout of image region with two image boundaries (upper and lower boundary). The final saliency map of 360-degree image is calculated by fusing feature contrast map and boundary connectivity map. Experimental results on a public eye tracking database of 360-degree images show promising performance of saliency prediction from the proposed method.

Perceptually Improved 3D Object Representation Based on Guided Adaptive Weighting of Feature Channels of a Visual-Attention Model

Real-time interaction in virtual environments composed of numerous objects modeled with a high number of faces remains an important issue in interactive virtual environment applications. A well-established approach to deal with this problem is to simplify small or distant objects where minor details are not informative for users. Several approaches exist in literature to simplify a 3D mesh uniformly. A possible improvement to this approach is to take advantage of a visual attention model to distinguish regions of a model which are considered important from the point of view of the human visual system. These regions can then be preserved during simplification to improve the perceived quality of the model. In the present article, we present an original application of biologically-inspired visual attention for improved perception-based representation of 3D objects. An enhanced visual attention model extracting information about color, intensity, orientation, as in the classical bottom-up visual attention model, but that also considers supplementary features believed to guide the deployment of human visual attention (such as symmetry, curvature, contrast, entropy and edge information), is introduced to identify such salient regions. Unlike the classical model where these features contribute equally to the identification of salient regions, a novel solution is proposed to adjust their contribution to the visual-attention model based on their compliance with points identified as salient by human subjects. An iterative approach is then proposed to extract salient points from salient regions. Salient points derived from images taken from best viewpoints of a 3D object are then projected to the surface of the object to identify salient vertices which will be preserved in the mesh simplification. The obtained results are compared with existing solutions from the literature to demonstrate the superiority of the proposed approach.

Extraction of coastal raft cultivation area with heterogeneous water background by thresholding object-based visually salient NDVI from high spatial resolution imagery

ABSTRACTThe development of high spatial resolution satellite imaging has enabled the acquisition of mariculture area information. This data could play an important role in mariculture investigations, ocean disaster evaluations, and coastal management. Because chlorophyll is concentrated in the widely distributed raft culture (a major kind of mariculture), the Normalized Difference Vegetation Index (NDVI) can be used for extraction. However, extensive coastal raft culture is easily confused with the heterogeneous water background. This results in unsatisfactory extraction when surveying a large water area with heterogeneous water background. By combining object-based image analysis and the centre-surround mechanism of a visual attention model, we propose an object-based visually salient NDVI (OBVS-NDVI) feature. Comparison experiments using Gaofen-2 spectral imagery of Luoyuan Bay, Fuzhou, China, indicate that OBVS-NDVI can effectively discriminate raft cultivation areas over large areas with a heterogeneous water background.

Vulnerable Plaque Recognition Based on Attention Model with Deep Convolutional Neural Network.

Previous studies have proved that the vulnerable plaque is a major factor leading to the onset of acute coronary syndrome (ACS). Recognizing vulnerable plaques is essential for cardiologists to treat illnesses, early. However, this task often comes with the challenge of insufficient annotated data sets and subtle differences between lesion regions and normal regions. In this paper, we apply the visual attention model with deep neural network to improve the performance of recognizing vulnerable plaques. There are two key ideas about our method: 1) using a top-down attention model to extract salient regions (blood vessels) according to the doctor's prior knowledge, and 2) employing a multi-task neural network to complete the recognition task. The first branch, a typical classification task, is to distinguish whether the image contains vulnerable plaques. The other branch uses a column-wise segmentation to locate vulnerable plaques. We have verified the effectiveness of our proposed method on the data set provided by 2017 CCCV-IVOCT Challenge. The proposed method obtains good performance.

Action Recognition by an Attention-Aware Temporal Weighted Convolutional Neural Network.

Research in human action recognition has accelerated significantly since the introduction of powerful machine learning tools such as Convolutional Neural Networks (CNNs). However, effective and efficient methods for incorporation of temporal information into CNNs are still being actively explored in the recent literature. Motivated by the popular recurrent attention models in the research area of natural language processing, we propose the Attention-aware Temporal Weighted CNN (ATW CNN) for action recognition in videos, which embeds a visual attention model into a temporal weighted multi-stream CNN. This attention model is simply implemented as temporal weighting yet it effectively boosts the recognition performance of video representations. Besides, each stream in the proposed ATW CNN framework is capable of end-to-end training, with both network parameters and temporal weights optimized by stochastic gradient descent (SGD) with back-propagation. Our experimental results on the UCF-101 and HMDB-51 datasets showed that the proposed attention mechanism contributes substantially to the performance gains with the more discriminative snippets by focusing on more relevant video segments.

Multi-scale contrast and relative motion-based key frame extraction

The huge amount of video data available these days requires effective management techniques for storage, indexing, and retrieval. Video summarization, a method to manage video data, provides concise versions of the videos for efficient browsing and retrieval. Key frame extraction is a form of video summarization which selects only the most salient frames from a given video. Since the automatic semantic understanding of the video contents is not possible so far, most of the existing works employ low level index features for extracting key frames. However, the usage of low level features results in loss of semantic details, thus leading to a semantic gap. In this context, the saliency-based user attention modeling technique can be used to bridge this semantic gap. In this paper, a key frame extraction scheme based on a visual attention mechanism is proposed. The proposed scheme builds static visual attention method based on multi-scale contrast instead of usual color contrast. The dynamic visual attention model is developed based on novel relative motion intensity and relative motion orientation. An efficient fusion scheme for combining three visual attention values is then proposed. A flexible technique is then used for key frame extraction. The experimental results demonstrate that the proposed mechanism provides excellent results as compared to the some of the other prominent techniques in the literature.

Top-down contributions to attention shifting and disengagement: A template model of visual attention.

Two separate systems are involved in the control of spatial attention; one that is driven by a goal, and the other that is driven by stimuli. While the goal- and stimulus-driven systems follow different general principles, they also interplay with each other. However, the mechanism by which the goal-driven system influences the stimulus-driven system is still debated. The present study examined top-down contributions to two components of attention orienting, shifting and disengagement, with an experimental paradigm in which participants held a visual item in short-term memory (STM) and performed a prosaccade task with a manipulation of the gap between fixation offset and target onset. Four experiments showed that the STM content accelerated shifting and impaired disengagement, but the influence on disengagement depended on the utility of STM in guiding attention toward the target. Thus, the use of STM was strategic. Computational models of visual attention were fitted to the experimental data, which suggested that the top-down contributions to shifting was more prominent than those to disengagement. The results indicate that the current modeling framework was particularly useful when examining the contributions of theoretical constructs for the control of visual attention, but it also suggests limitations. (PsycINFO Database Record

Toolbox and dataset for the development of saliency and scanpath models for omnidirectional/360° still images

Research on understanding how people watch/explore Virtual Reality (VR) and 360° content is crucial to develop appropriate technologies for processing, encoding, delivering and rendering media content in order to provide high-quality immersive experiences to users. In this sense, the work described in this paper aims at supporting the research on visual attention modeling for omnidirectional content, by presenting a dataset and toolbox that were created for the Grand Challenge “Salient360!” organized at ICME’17. In particular, the dataset includes 360° images and their corresponding saliency maps and scanpaths, obtained from a subjective experiment in which eye gazes and head movements of observers were recorded during a free-viewing task. In addition, guidelines to obtain saliency maps and scanpaths from eye and head tracking data for omnidirectional images are provided. Finally, together with the dataset, a toolbox has been made publicly available, containing scripts for comparing saliency maps and scanpaths to benchmark developed models.

Visual attention prediction for images with leading line structure

Researchers have proposed a wide variety of visual attention models, ranging from models that use local, low-level image features to recent approaches that incorporate semantic information. However, most models do not account for the visual attention evident in images with certain global structures. We focus specifically on “leading line” structures, in which explicit or implicit lines converge at a point. Through this study, we have conducted the experiments to investigate the visual attentions in images with leading line structure and propose new models that combine the low-level feature of center-surround differences of visual stimuli, the semantic feature of center bias and the structure feature of leading lines. We also create a new data set from 110 natural images containing leading lines and the eye-tracking data for 16 subjects. Our evaluation experiment showed that our models outperform the existing models against common indicators of saliency-map evaluation, underscoring the importance of leading lines in the modeling of visual attention.

Object recognition in clutter color images using Hierarchical Temporal Memory combined with salient-region detection

Abstract The essential goal of this paper is to extend the functionality of the bio-inspired intelligent HTM (Hierarchical Temporal Memory) network towards two capabilities: (i) object recognition in color images, (ii) detection of multiple objects located in clutter color images. The former extension is based on the development of a novel scheme for the application of three parallel HTM networks that separately process color, texture, and shape information in color images. For the latter HTM extension, we propose a novel system in which HTM is combined with a modified model of computational visual attention. We adopt the results of Bi et al. (2010), Hu et al. (2005), and Kucerova (2011) and add new elements for the calculation of image saliency maps. The proposed algorithm enables one to automatically locate individual objects in clutter images. For computer experiments, a special image database is created to simulate ideal single object images and cluttered images with multiple objects on an inhomogeneous background. The recognition performance of HTM alone and in combination with the salient-region detection method is evaluated. We show that the attention subsystem is able to satisfactorily locate multiple objects in clutter color images with an inhomogeneous background. We also perform benchmark calculations for two selected computer vision methods used for object detection in color clutter images. Namely, the cascade detector and template matching methods are used. Our study confirms that the proposed attention system can improve the capabilities of HTM for object classification in cluttered images. The compound system of visual attention and HTM outperforms the compared methods in both criteria (recall and correct detection rate). However, as expected, the system cannot match the recognition accuracy achieved by HTM for single object images, and thus, further research is needed.

A COARSE-TO-FINE MODEL FOR AIRPLANE DETECTION FROM LARGE REMOTE SENSING IMAGES USING SALIENCY MODLE AND DEEP LEARNING

Abstract. High resolution remote sensing images are bearing the important strategic information, especially finding some time-sensitive-targets quickly, like airplanes, ships, and cars. Most of time the problem firstly we face is how to rapidly judge whether a particular target is included in a large random remote sensing image, instead of detecting them on a given image. The problem of time-sensitive-targets target finding in a huge image is a great challenge: 1) Complex background leads to high loss and false alarms in tiny object detection in a large-scale images. 2) Unlike traditional image retrieval, what we need to do is not just compare the similarity of image blocks, but quickly find specific targets in a huge image. In this paper, taking the target of airplane as an example, presents an effective method for searching aircraft targets in large scale optical remote sensing images. Firstly, we used an improved visual attention model utilizes salience detection and line segment detector to quickly locate suspected regions in a large and complicated remote sensing image. Then for each region, without region proposal method, a single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation is adopted to search small airplane objects. Unlike sliding window and region proposal-based techniques, we can do entire image (region) during training and test time so it implicitly encodes contextual information about classes as well as their appearance. Experimental results show the proposed method is quickly identify airplanes in large-scale images.

Objective evaluation of fabric pilling based on data-driven visual attention model

PurposeThe purpose of this paper is to develop a new objective evaluation method of fabric pilling using data-driven visual attention model.Design/methodology/approachFirst, the multi-scale filtering images are formed by Gaussian pyramid decomposition. Second, center-surround differences algorithm is used between multi-scale filtering images to build saliency map. On this basis, the pilling information is segmented from saliency map by the segmentation threshold. Finally, the pilling is objectively evaluated by extracting pilling feature. Experimental result shows that compared with the traditional detection methods, the proposed objective evaluation method has strong anti-interference ability, and correct classification rate (CCR) is 96 percent.FindingsFabric pilling saliency can be effectively improved by data-driven visual attention model, which will lead to stronger anti-interference ability and higher correct classification rate.Originality/valueTo void uneven illumination, noise, and texture interference, the proposed method can enhance the saliency of small targets in saliency map using a bottom-up visual attention model. Through the threshold segmentation according to pilling feature, the pilling information is effectively from the fabric texture. Pilling feature about pilling area density is extracted to pilling grade evaluation.

Visual Attention-Based Target Detection and Discrimination for High-Resolution SAR Images in Complex Scenes

The conventional methods for target detection and discrimination in high-resolution synthetic aperture radar (SAR) images usually have low accuracy and slow speed, especially for large complex scenes. To overcome these drawbacks, in this paper, we propose a target detection and discrimination method based on visual attention model. In the detection stage, to pop out the targets and suppress the background clutter in the saliency map, we select the task-dependent scales from the Gaussian pyramid of the original SAR image. Moreover, we adopt the clustering algorithm to remerge several isolated focus of attention areas, which are obtained from the saliency map, into a complete target region. The candidate target SAR image chips are extracted with relative high accuracy and low time cost in this stage. Since there may be single target, multiple targets, or partial targets with complex clutter in each SAR image chip, it is hard to acquire accurate target-shaped blob via segmentation. Some classical discrimination features which are extracted based on target segmentation may lose effectiveness. In the discrimination stage of our method, to solve the above problem, based on the saliency and gist (SG) features for optical satellite images, we propose the modified SG (MSG) features for SAR target discrimination. The MSG features are complementary to each other and can provide a more complete description of the extracted SAR image chips without segmentation, which also reduces the computation burden. The experimental results on the synthetic images and miniSAR real SAR image data set demonstrate that the proposed target detection and discrimination method can detect and discriminate the targets from the complex background clutter with high accuracy and fast speed in high-resolution SAR images.

Saliency in VR: How Do People Explore Virtual Environments?

Understanding how people explore immersive virtual environments is crucial for many applications, such as designing virtual reality (VR) content, developing new compression algorithms, or learning computational models of saliency or visual attention. Whereas a body of recent work has focused on modeling saliency in desktop viewing conditions, VR is very different from these conditions in that viewing behavior is governed by stereoscopic vision and by the complex interaction of head orientation, gaze, and other kinematic constraints. To further our understanding of viewing behavior and saliency in VR, we capture and analyze gaze and head orientation data of 169 users exploring stereoscopic, static omni-directional panoramas, for a total of 1980 head and gaze trajectories for three different viewing conditions. We provide a thorough analysis of our data, which leads to several important insights, such as the existence of a particular fixation bias, which we then use to adapt existing saliency predictors to immersive VR conditions. In addition, we explore other applications of our data and analysis, including automatic alignment of VR video cuts, panorama thumbnails, panorama video synopsis, and saliency-basedcompression.

A Method of Objects Classification Based on Learning and Visual Perception

Objects classification is one of the most significant problems in computer vision. For improving the accuracy of objects classification, we put forward a new classification method enlightened the whole process that human distinguish different types of objects. Our method mixed visual saliency model and CNN, is more close to human and has apparently biological advantages. Firstly, we built an eye-tracking database to learn people visual behaviors when they classify various objects and recorded the eye-tracking data. Secondly, this database is used to train a learning-based visual attention model, which is based on low-level (e.g., orientation, color, intensity, etc.) and high-level (e.g., faces, people, cars, etc.) image features to analyze and predict the human's classification RoIs. Finally, we established a CNN framework to classify RoIs. The results of the experiment showed our attention model can determine saliency regions and predict human's classification RoIs more precisely and our classification method improved the efficiency of classification markedly.

GBVS360, BMS360, ProSal: Extending existing saliency prediction models from 2D to omnidirectional images

In this paper it is studied how existing visual saliency models designed for 2D images can be extended and applied to Omnidirectional images in the equirectangular format. Three different frameworks, BMS360, GBVS360 and Projected Saliency were designed to address this task. In the particular case of BMS360 and GBVS360, the 2D models Boolean Map Saliency (BMS) and Graph-Based Visual Saliency (GBVS) were rigorously extended to integrate the specific properties of equirectangular images to model visual attention in omnidirectional images watched using Head Mounted Displays (HMDs). With the proposed extensions, the saliency prediction performance in comparison to their original design could significantly be improved. Another key result of this paper comes from the design of a framework called projected saliency which allows applying existing saliency models on equirectangular images without requiring in-depth adaptation of the prediction algorithms. The projected saliency framework enabled to study the interaction of features in the feature activation process of computational saliency models. It shows that the activation should not be done per-viewport, but should account for neighboring regions. As a consequence, we argue that the feature activation process should be performed at a global level instead of a local one. Here, the use of a local approach with a large FOV was found to be a good compromise between global activation and appropriate feature computation to directly port existing saliency models designed for rectilinear images to the domain of equirectangular images. In this paper an adaptive equatorial prior is also described. This equatorial prior allows to account for pitch shift of the horizon line. Finally, results show that the newly designed BMS360 and GVBS360 outperform even recent state-of-the-art models at head motion prediction and are competitive at head/eye saliency-map prediction. Moreover, all models and their code are made publicly available on GitHub allowing to further iterate on this work.

A feature-based approach for saliency estimation of omni-directional images

Omni-directional imaging records the visual information from any direction with respect to a given view-point. It is gaining consumers’ popularity due to fast spreading of low-cost devices both for acquisition and rendering. The possibility to render the whole surrounding space represents a further step towards immersivity, thus providing the user with the illusion of physically being in a virtual environment. The understanding of visual attention mechanisms for these images is a relevant topic for processing, coding, and exploiting such data. In this contribution, a saliency model for omni-directional images is presented. It is based on the combination of low-level and semantic features. The first ones account for texture, viewport saliency, hue and saturation, while the second are used to take into account the impact of the presence of human subjects on the saliency. The proposed model has been tested in the “Salient360! Visual attention modeling for 360° Images” Grand Challenge. The model, the achieved results, and finding/discussions are here presented.