Saliency-based Visual Attention Research Articles

An event-based implementation of saliency-based visual attention for rapid scene analysis

An event-based implementation of saliency-based visual attention for rapid scene analysis

Depth-aware salient object segmentation

Depth-aware salient object segmentation

Research on Region Selection Strategy for Visible Watermark Embedding

ABSTRACT Visible watermarking is one of the most direct and effective copyright protection methods for digital images and videos, but it faces the problems of easy detection of watermarking location and watermarking removal attacks. It requires that the visible watermarking not only can not affect visual quality of original digital products but also can resist accidental editing and malicious attacks. In this paper, an adaptive region selection method for visible watermark embedding is proposed. First, an improved region selection method based on the model of saliency-based visual attention (MSVA) is used to more accurately distinguish salient and non-salient regions of host image; then, the non-salient regions of image are divided into sub-blocks to calculate texture complexity respectively; finally, a moderate image block is selected as the embedding region of visible watermark image according to the distribution of image texture complexity. The experimental results show that the proposed method can identify saliency regions more accurately, thus ensuring that the important contents of host image will not be occluded. The adaptive selection of embedding region based on image complexity not only ensures the recognizability and aesthetics of visible watermark embedding effect but also takes into account the security of watermark embedding, and especially reduces the risk of batch removal attacks.

Capturing what human eyes perceive: A visual hierarchy generation approach to emulating saliency-based visual attention for grid-like urban street networks

Visual hierarchy is an important notion in urban imagery research. As the skeletons of cities, urban streets attract more attention from urban residents and street network hierarchies are important references for urban planning and urban studies. However, due to the characteristic of over-regularization, it is often difficult for humans to differentiate visual salience for grid-like street networks, resulting in the hierarchies of grid-like streets yielded by existing methods being prone to cause visual cognitive confusion. Therefore, in this study, we proposed a novel model to quantify the extent to which a street attracts human visual attention through emulating the visual attention mechanism that can capture the focus of relatively significant elements at different levels of perception. Using the natural street (also known as the stroke) as the sensor unit, the comprehensive visual salience (CVS) index combining the geometric competitive factors of natural streets at the local scale and psychological competitive factors of natural streets at the global scale is designed. Finally, the visual salience of the urban natural streets is ranked by these CVS scores and the visual hierarchy is derived by the head/tail breaks scheme. The model was applied to eight typical grid-like street networks and the results show that the performance of visual discrimination on street hierarchies is greatly improved. Our hierarchy generation method could effectively detect visually prominent streets for grid-like street networks and generate the visual hierarchies of grid-like street networks that conform to the hierarchies perceived by human eyes. These results would provide helpful suggestions in practical urban street network applications.

Multiple rotation symmetry group detection via saliency-based visual attention and Frieze expansion pattern

Global maximum symmetry center probability-based rotation symmetry detection methods are unable to identify small-scale rotation symmetry centers in real-world images and come with a costly computational burden. This paper presents a novel strategy comprised of multiple-local-region searching in the whole image and multiple-model computation to map rotation symmetry strength (RSS) in the local region. The multiple-local-region searching method creates local regions in which the global maximum symmetry center detection method changes to a local maximum symmetry center detection method. In the local region, the multiple-model computation efficiently detects regular, small-scale, and skewed symmetry centers. This strategy improves the detection ability for regular, small-scale, and skewed rotation symmetry centers while minimizing the complexity of the algorithm based on a rotation symmetry strength (RSS) map and symmetry shape density (SSD) map. Experimental results indicate that this strategy not only allows more rotation symmetry centers to be identified successfully, but is simpler than traditional strategies as it only employs RSS maps.

Distorted Low-Level Visual Features Affect Saliency-Based Visual Attention.

Image distortions can attract attention away from the natural scene saliency (Redi et al., 2011). Performance of viewers in visual search tasks and their fixation patterns are also affected by different types and amounts of distortions (Vu et al., 2008). In this paper, we have discussed the opinion that distortions could largely affect the performance of predictive models of visual attention, and simulated the effects of distorted low-level visual features on the saliency-based bottom-up visual attention. Saliency is a fast and pre-attentive mechanism for orienting visual attention to intrinsically important objects which pop-out more easily in a cluttered scene. Distortion of the low-level features that contribute to saliency may impair the readiness of the visual system in detection of salient objects, which may have major implications for critical situations like driving or locomotion. These distortions in natural life can be introduced by eye diseases such as cataract, or spectacles which may alter color perception (de Fez et al., 2002) or cause undesired optical effects like blurring, non-uniform magnification, and image displacement (Barbero and Portilla, 2016). The extent to which each of these distorted saliency features may affect the attentional performance is addressed in this paper by employing a biologically-inspired predictive model of visual attention. We briefly summarize the current standing of computational work on visual attention models in the following section and suggest a simple and influential model of saliency to examine the above hypothesis. Furthermore, we demonstrate in an example the hindered performance of the predictive saliency model on distorted images.

Saliency-based gaze prediction based on head direction

Despite decades of attempts to create a model for predicting gaze locations by using saliency maps, a highly accurate gaze prediction model for general conditions has yet to be devised. In this study, we propose a gaze prediction method based on head direction that can improve the accuracy of any model. We used a probability distribution of eye position based on head direction (static eye–head coordination) and added this information to a model of saliency-based visual attention. Using empirical data on eye and head directions while observers were viewing natural scenes, we estimated a probability distribution of eye position. We then combined the relationship between eye position and head direction with visual saliency to predict gaze locations. The model showed that information on head direction improved the prediction accuracy. Further, there was no difference in the gaze prediction accuracy between the two models using information on head direction with and without eye–head coordination. Therefore, information on head direction is useful for predicting gaze location when it is available. Furthermore, this gaze prediction model can be applied relatively easily to many daily situations such as during walking.

Autonomous Detection of Weld Seam Profiles via a Model of Saliency-Based Visual Attention for Robotic Arc Welding

This paper presents a method of autonomously detecting weld seam profiles from molten pool background in metal active gas (MAG) arc welding using a novel model of saliency-based visual attention. First, a vision sensor based on structured light is employed to capture laser stripes and molten pools simultaneously in the same frame. Second, to effectively detect the weld seam profile from molten pool background for next autonomous guidance of initial welding positions and seam tracking, a model of visual attention based on saliency is proposed. With respect to the enhanced effect of saliency, the proposed model is much better than the classic models in the field. According to the comprehensive saliency map created by the proposed model, the weld seam profile can be extracted after threshold segmentation and clustering are applied to it in turn. Third, different weld seam images are used to demonstrate the robustness of the proposed methodology and last, to evaluate the performance of the proposed method, a measure called profile extraction rate (PER) is computed, which shows that the extracted weld seam profile can basically meet the requirements of seam tracking and the guidance of welding torches.

Recognition of suspicious behavior using case-based reasoning

A novel method case-based reasoning was proposed for suspicious behavior recognition. The method is composed of three departs: human behavior decomposition, human behavior case representation and case-based reasoning. The new approach was proposed to decompose behavior into sub-behaviors that are easier to recognize using a saliency-based visual attention model. New representation of behavior was introduced, in which the sub-behavior and the associated time characteristic of sub-behavior were used to represent behavior case. In the process of case-based reasoning, apart from considering the similarity of basic sub-behaviors, order factor was proposed to measure the similarity of a time order among the sub-behaviors and span factor was used to measure the similarity of duration time of each sub-behavior, which makes the similarity calculations more rational and comprehensive. Experimental results show the effectiveness of the proposed method in comparison with other related works and can run in real-time for the recognition of suspicious behaviors.

Biological edge detection for UCAV via improved artificial bee colony and visual attention

Purpose – The purpose of this paper is to propose a biological edge detection approach for aircraft such as unmanned combat air vehicle (UCAV), with the objective of making the UCAV recognize targets, especially in complex noisy environment. Design/methodology/approach – The hybrid model of saliency-based visual attention and artificial bee colony (ABC) algorithm is established for edge detection of UCAV. Visual attention can extract the region of interesting objects, and this approach can narrow the searching region for object segmentation, which can reduce the computational complexity. An improved ABC algorithm is applied in edge detection of the salient region. Findings – This work improved ABC algorithm by modifying the search strategy and adding some limits, so that it can be applied to edge detection problem. A hybrid model of saliency-based visual attention and ABC algorithm is developed. Experimental results demonstrated the feasibility and effectiveness of the proposed method: it can guarantee ef...

A New Approach of Feature Combination for Object Detection in Saliency-based Visual Attention

This paper presents a fuzzy approach of feature maps combination in saliency-based visual attention model proposed by Itti. This strategy applies fuzzy rules to combine three conspicuity maps instead of linear combination in the basic model of visual attention that does not seem reasonable biologically. In this method, in addition to bottom-up features, top-down cues are also considered in the model. As fuzzy rules are designed using target mask information, top-down characteristics of the target are considered helping the model to make the target more conspicuous in the final saliency map. This can be applied in further processing such as object detection and recognition application. The experimental results show the effectiveness of our new fuzzy approach in finding the target in the first hit. A database of emergency triangle in natural environment background is used in this paper to show the results. Moreover, the comparison of this fuzzy combination approach with some other combination methods also proved the priority of the approach over other combination strategies.

Color coding in the cortex: a modified approach to bottom-up visual attention

Itti and Koch's (Vision Research 40:1489-1506, 2000) saliency-based visual attention model is a broadly accepted model that describes how attention processes are deployed in the visual cortex in a pure bottom-up strategy. This work complements their model by modifying the color feature calculation. Evidence suggests that S-cone responses are elicited in the same spatial distribution and have the same sign as responses to M-cone stimuli; these cells are tentatively referred to as red-cyan. For other cells, the S-cone input seems to be aligned with the L-cone input; these cells might be green-magenta cells. To model red-cyan and green-magenta double-opponent cells, we implement a center-surround difference approach of the aforementioned model. The resulting color maps elicited enhanced responses to color salient stimuli when compared to the classic ones at high statistical significance levels. We also show that the modified model improves the prediction of locations attended by human viewers.

Learning saliency-based visual attention: A review

Humans and other primates shift their gaze to allocate processing resources to a subset of the visual input. Understanding and emulating the way that human observers free-view a natural scene has both scientific and economic impact. It has therefore attracted the attention from researchers in a wide range of science and engineering disciplines. With the ever increasing computational power, machine learning has become a popular tool to mine human data in the exploration of how people direct their gaze when inspecting a visual scene. This paper reviews recent advances in learning saliency-based visual attention and discusses several key issues in this topic.

Fast Visual Attention Model Algorithm Based on Approximate Gaussian Pyramids

Classical saliency-based visual attention models are adapted for embedding real-time systems with less time and space costs based on approximate Gaussian pyramids of the input image. Firstly, the circular window and discrete Gaussian convolution are approximated by rectangular window and rectangular average operator respectively. Then, rectangular average operator is implemented through "row accumulation followed by column accumulation". And conspicuity maps of each channel are calculated and sampled at desired intervals directly with linear computational complexity on the number of the input pixels. At last, a fast algorithm for inhibiting the saliency of extracted regions in the saliency map is proposed. Experimental results in the images from Berkeley segmentation dataset validate that the proposed methods have much less computational costs with acceptable outputting errors. The two approximate methods in this paper can also be applied in other image processing problems in embedding real-time systems.

Predicting visual fixations on video based on low-level visual features

To what extent can a computational model of the bottom–up visual attention predict what an observer is looking at? What is the contribution of the low-level visual features in the attention deployment? To answer these questions, a new spatio-temporal computational model is proposed. This model incorporates several visual features; therefore, a fusion algorithm is required to combine the different saliency maps (achromatic, chromatic and temporal). To quantitatively assess the model performances, eye movements were recorded while naive observers viewed natural dynamic scenes. Four completing metrics have been used. In addition, predictions from the proposed model are compared to the predictions from a state of the art model [Itti’s model (Itti, L., Koch, C., & Niebur, E. (1998). A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(11), 1254–1259)] and from three non-biologically plausible models (uniform, flicker and centered models). Regardless of the metric used, the proposed model shows significant improvement over the selected benchmarking models (except the centered model). Conclusions are drawn regarding both the influence of low-level visual features over time and the central bias in an eye tracking experiment.

Bottom-up spatiotemporal visual attention model for video analysis

The human visual system (HVS) has the ability to fixate quickly on the most informative (salient) regions of a scene and therefore reducing the inherent visual uncertainty. Computational visual attention (VA) schemes have been proposed to account for this important characteristic of the HVS. A video analysis framework based on a spatiotemporal VA model is presented. A novel scheme has been proposed for generating saliency in video sequences by taking into account both the spatial extent and dynamic evolution of regions. To achieve this goal, a common, image-oriented computational model of saliency-based visual attention is extended to handle spatiotemporal analysis of video in a volumetric framework. The main claim is that attention acts as an efficient preprocessing step to obtain a compact representation of the visual content in the form of salient events/objects. The model has been implemented, and qualitative as well as quantitative examples illustrating its performance are shown.

Feature combination strategies for saliency-based visual attention systems

Bottom-up or saliency-based visual attention allows primates to detect nonspecific conspicuous targets in cluttered scenes. A classical metaphor, derived from electrophysiological and psychophysical studies, describes attention as a rapidly shiftable “spotlight.” We use a model that reproduces the attentional scan paths of this spotlight. Simple multi-scale “feature maps” detect local spatial discontinuities in intensity, color, and orientation, and are combined into a unique “master” or “saliency” map. The saliency map is sequentially scanned, in order of decreasing saliency, by the focus of attention. We here study the problem of combining feature maps, from different visual modalities (such as color and orientation), into a unique saliency map. Four combination strategies are compared using three databases of natural color images: (1) Simple normalized summation, (2) linear combination with learned weights, (3) global nonlinear normalization followed by summation, and (4) local nonlinear competition between salient locations followed by summation. Performance was measured as the number of false detections before the most salient target was found. Strategy (1) always yielded poorest performance and (2) best performance, with a threefold to eightfold improvement in time to find a salient target. However, (2) yielded specialized systems with poor generalization. Interestingly, strategy (4) and its simplified, computationally efficient approximation (3) yielded significantly better performance than (1), with up to fourfold improvement, while preserving generality.

A model of saliency-based visual attention for rapid scene analysis

A visual attention system, inspired by the behavior and the neuronal architecture of the early primate visual system, is presented. Multiscale image features are combined into a single topographical saliency map. A dynamical neural network then selects attended locations in order of decreasing saliency. The system breaks down the complex problem of scene understanding by rapidly selecting, in a computationally efficient manner, conspicuous locations to be analyzed in detail.