Bottom-up Attention Model Research Articles

Event-driven proto-object based saliency in 3D space to attract a robot’s attention

To interact with its environment, a robot working in 3D space needs to organise its visual input in terms of objects or their perceptual precursors, proto-objects. Among other visual cues, depth is a submodality used to direct attention to visual features and objects. Current depth-based proto-object attention models have been implemented for standard RGB-D cameras that produce synchronous frames. In contrast, event cameras are neuromorphic sensors that loosely mimic the function of the human retina by asynchronously encoding per-pixel brightness changes at very high temporal resolution, thereby providing advantages like high dynamic range, efficiency (thanks to their high degree of signal compression), and low latency. We propose a bio-inspired bottom-up attention model that exploits event-driven sensing to generate depth-based saliency maps that allow a robot to interact with complex visual input. We use event-cameras mounted in the eyes of the iCub humanoid robot to directly extract edge, disparity and motion information. Real-world experiments demonstrate that our system robustly selects salient objects near the robot in the presence of clutter and dynamic scene changes, for the benefit of downstream applications like object segmentation, tracking and robot interaction with external objects.

Awareness-Dependent Normalization Framework of Visual Bottom-up Attention.

Although bottom-up attention can improve visual performance with and without awareness to the exogenous cue, whether they are governed by a common neural computation remains unclear. Using a modified Posner paradigm with backward masking, we found that the cueing effect displayed a monotonic gradient profile (Gaussian-like), both with and without awareness, whose scope, however, was significantly wider with than without awareness. This awareness-dependent scope offered us a unique opportunity to change the relative size of the attention field to the stimulus, differentially modulating the gain of attentional selection, as proposed by the normalization model of attention. Therefore, for each human subject (male and female), the stimulus size was manipulated as their respective mean attention fields with and without awareness while stimulus contrast was varied in a spatial cueing task. By measuring the gain pattern of contrast-response functions on the spatial cueing effect derived by visible or invisible cues, we observed changes in the cueing effect consonant with changes in contrast gain for visible cues and response gain for invisible cues. Importantly, a complementary analysis confirmed that subjects' awareness-dependent attention fields can be simulated by using the normalization model of attention. Together, our findings indicate an awareness-dependent normalization framework of visual bottom-up attention, placing a necessary constraint, namely, awareness, on our understanding of the neural computations underlying visual attention.SIGNIFICANCE STATEMENT Bottom-up attention is known to improve visual performance with and without awareness. We discovered that manipulating subjects' awareness can modulate their attention fields of visual bottom-up attention, which offers a unique opportunity to regulate its normalization processes. On the one hand, by measuring the gain pattern of contrast-response functions on the spatial cueing effect derived by visible or invisible cues, we observed changes in the cueing effect consonant with changes in contrast gain for visible cues and response gain for invisible cues. On the other hand, by using the normalization model of attention, subjects' awareness-dependent attention fields can be simulated successfully. Our study supports important predictions of the normalization model of visual bottom-up attention and further reveals its dependence on awareness.

Attention-based active visual search for mobile robots

We present an active visual search model for finding objects in unknown environments. The proposed algorithm guides the robot towards the sought object using the relevant stimuli provided by the visual sensors. Existing search strategies are either purely reactive or use simplified sensor models that do not exploit all the visual information available. In this paper, we propose a new model that actively extracts visual information via visual attention techniques and, in conjunction with a non-myopic decision-making algorithm, leads the robot to search more relevant areas of the environment. The attention module couples both top-down and bottom-up attention models enabling the robot to search regions with higher importance first. The proposed algorithm is evaluated on a mobile robot platform in a 3D simulated environment. The results indicate that the use of visual attention significantly improves search, but the degree of improvement depends on the nature of the task and the complexity of the environment. In our experiments, we found that performance enhancements of up to 42% in structured and 38% in highly unstructured cluttered environments can be achieved using visual attention mechanisms.

Combining segmentation and attention: a new foveal attention model.

Artificial vision systems cannot process all the information that they receive from the world in real time because it is highly expensive and inefficient in terms of computational cost. Inspired by biological perception systems, artificial attention models pursuit to select only the relevant part of the scene. On human vision, it is also well established that these units of attention are not merely spatial but closely related to perceptual objects (proto-objects). This implies a strong bidirectional relationship between segmentation and attention processes. While the segmentation process is the responsible to extract the proto-objects from the scene, attention can guide segmentation, arising the concept of foveal attention. When the focus of attention is deployed from one visual unit to another, the rest of the scene is perceived but at a lower resolution that the focused object. The result is a multi-resolution visual perception in which the fovea, a dimple on the central retina, provides the highest resolution vision. In this paper, a bottom-up foveal attention model is presented. In this model the input image is a foveal image represented using a Cartesian Foveal Geometry (CFG), which encodes the field of view of the sensor as a fovea (placed in the focus of attention) surrounded by a set of concentric rings with decreasing resolution. Then multi-resolution perceptual segmentation is performed by building a foveal polygon using the Bounded Irregular Pyramid (BIP). Bottom-up attention is enclosed in the same structure, allowing to set the fovea over the most salient image proto-object. Saliency is computed as a linear combination of multiple low level features such as color and intensity contrast, symmetry, orientation and roundness. Obtained results from natural images show that the performance of the combination of hierarchical foveal segmentation and saliency estimation is good in terms of accuracy and speed.

Efficient object detection based on selective attention

In this paper, we make use of biologically inspired selective attention to improve the efficiency and performance of object detection under clutter. At first, we propose a novel bottom-up attention model. We argue that heuristic feature selection based on bottom-up attention can stably select out invariant and discriminative features. With these selected features, performance of object detection can be improved apparently and stably. Then we propose a novel concept of saccade map based on bottom-up attention to simulate the saccade (eye movements) in vision. Sliding within saccade map to detect object can significantly reduce computational complexity and apparently improve performance because of the effective filtering for distracting information. With these ideas, we present a general framework for object detection through integrating bottom-up attention. Through evaluating on UIUC cars and Weizmann–Shotton horses we show state-of-the-art performance of our object detection model.

Development and Evaluation of Object-Based Visual Attention for Automatic Perception of Robots

Bottom-up visual attention is an automatic behavior to guide visual perception to a conspicuous object in a scene. This paper develops a new object-based bottom-up attention (OBA) model for robots. This model includes four modules: Extraction of preattentive features, preattentive segmentation, estimation of space-based saliency, and estimation of proto-object-based saliency. In terms of computation, preattentive segmentation serves as a bridge to connect the space-based saliency and object-based saliency. This paper therefore proposes a preattentive segmentation algorithm, which is able to self-determine the number of proto-objects, has low computational cost, and is robust in a variety of conditions such as noise and spatial transformations. Experimental results have shown that the proposed OBA model outperforms space-based attention model and other object-based attention methods in terms of accuracy of attentional selection, consistency under a series of noise settings and object completion.

A saliency-based bottom-up visual attention model for dynamic scenes analysis

This work proposes a model of visual bottom-up attention for dynamic scene analysis. Our work adds motion saliency calculations to a neural network model with realistic temporal dynamics [(e.g., building motion salience on top of De Brecht and Saiki Neural Networks 19:1467-1474, (2006)]. The resulting network elicits strong transient responses to moving objects and reaches stability within a biologically plausible time interval. The responses are statistically different comparing between earlier and later motion neural activity; and between moving and non-moving objects. We demonstrate the network on a number of synthetic and real dynamical movie examples. We show that the model captures the motion saliency asymmetry phenomenon. In addition, the motion salience computation enables sudden-onset moving objects that are less salient in the static scene to rise above others. Finally, we include strong consideration for the neural latencies, the Lyapunov stability, and the neural properties being reproduced by the model.

A Time-Dependent Saliency Model Combining Center and Depth Biases for 2D and 3D Viewing Conditions

The role of the binocular disparity in the deployment of visual attention is examined in this paper. To address this point, we compared eye tracking data recorded while observers viewed natural images in 2D and 3D conditions. The influence of disparity on saliency, center and depth biases is first studied. Results show that visual exploration is affected by the introduction of the binocular disparity. In particular, participants tend to look first at closer areas in 3D condition and then direct their gaze to more widespread locations. Beside this behavioral analysis, we assess the extent to which state-of-the-art models of bottom-up visual attention predict where observers looked at in both viewing conditions. To improve their ability to predict salient regions, low-level features as well as higher-level foreground/background cues are examined. Results indicate that, consecutively to initial centering response, the foreground feature plays an active role in the early but also middle instants of attention deployments. Importantly, this influence is more pronounced in stereoscopic conditions. It supports the notion of a quasi-instantaneous bottom-up saliency modulated by higher figure/ground processing. Beyond depth information itself, the foreground cue might constitute an early process of “selection for action”. Finally, we propose a time-dependent computational model to predict saliency on still pictures. The proposed approach combines low-level visual features, center and depth biases. Its performance outperforms state-of-the-art models of bottom-up attention.

Decoding Successive Computational Stages of Saliency Processing

An important requirement for vision is to identify interesting and relevant regions of the environment for further processing. Some models assume that salient locations from a visual scene are encoded in a dedicated spatial saliency map [1, 2]. Then, a winner-take-all (WTA) mechanism [1, 2] is often believed to threshold the graded saliency representation and identify the most salient position in the visual field. Here we aimed to assess whether neural representations of graded saliency and the subsequent WTA mechanism can be dissociated. We presented images of natural scenes while subjects were in a scanner performing a demanding fixation task, and thus their attention was directed away. Signals in early visual cortex and posterior intraparietal sulcus (IPS) correlated with graded saliency as defined by a computational saliency model. Multivariate pattern classification [3, 4] revealed that the most salient position in the visual field was encoded in anterior IPS and frontal eye fields (FEF), thus reflecting a potential WTA stage. Our results thus confirm that graded saliency and WTA-thresholded saliency are encoded in distinct neural structures. This could provide the neural representation required for rapid and automatic orientation toward salient events in natural environments.

Hebbian-based neural networks for bottom-up visual attention and its applications to ship detection in SAR images

This paper proposes a bottom-up attention model based on pulsed Hebbian neural networks. The salience of the visual input can be generated through the networks using a simple normalization process, which can be calculated rapidly. Moreover, visual salience in this model can be represented as binary codes that mimic neuronal pulses in the human brain. Experimental results on psychophysical patterns and eye fixation prediction for natural images prove the effectiveness and efficiency of the model. In an arduous task of detecting ships in synthetic aperture radar (SAR) images, there are large amounts of data to be processed in real time. As a fast and effective technique for saliency detection, the proposed model is applied to ship detection in SAR images and its robustness against speckles is further proved.

Contour-Based Image Segmentation Using Selective Visual Attention

In many medical image segmentation applications identifying and extracting the region of interest (ROI) accurately is an important step. The usual approach to extract ROI is to apply image segmentation methods. In this paper, we focus on extracting ROI by segmentation based on visual attended locations. Chan-Vese active contour model is used for image segmentation and attended locations are determined by SaliencyToolbox. The implementation of the toolbox is extension of the saliency map-based model of bottom-up attention, by a process of inferring the extent of a proto-object at the attended location from the maps that are used to compute the saliency map. When the set of regions of interest is selected, these regions need to be represented with the highest quality while the remaining parts of the processed image could be represented with a lower quality. The method has been successfully tested on medical images and ROIs are extracted.

Attention via Synchrony: Making Use of Multimodal Cues in Social Learning

Infants learning about their environment are confronted with many stimuli of different modalities. Therefore, a crucial problem is how to discover which stimuli are related, for instance, in learning words. In making these multimodal ldquobindings,rdquo infants depend on social interaction with a caregiver to guide their attention towards relevant stimuli. The caregiver might, for example, visually highlight an object by shaking it while vocalizing the object's name. These cues are known to help structuring the continuous stream of stimuli. To detect and exploit them, we propose a model of bottom-up attention by multimodal signal-level synchrony. We focus on the guidance of visual attention from audio-visual synchrony informed by recent adult-infant interaction studies. Consequently, we demonstrate that our model is receptive to parental cues during child-directed tutoring. The findings discussed in this paper are consistent with recent results from developmental psychology but for the first time are obtained employing an objective, computational model. The presence of ldquomultimodal mothereserdquo is verified directly on the audio-visual signal. Lastly, we hypothesize how our computational model facilitates tutoring interaction and discuss its application in interactive learning scenarios, enabling social robots to benefit from adult-like tutoring.

Computational Analysis of Motionese Toward Scaffolding Robot Action Learning

A difficulty in robot action learning is that robots do not know where to attend when observing action demonstration. Inspired by human parent-infant interaction, we suggest that parental action demonstration to infants, called <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">motionese</i> , can scaffold robot learning as well as infants'. Since infants' knowledge about the context is limited, which is comparable to robots, parents are supposed to properly guide their attention by emphasizing the important aspects of the action. Our analysis employing a bottom-up attention model revealed that motionese has the effects of highlighting the initial and final states of the action, indicating significant state changes in it, and underlining the properties of objects used in the action. Suppression and addition of parents' body movement and their frequent social signals to infants produced these effects. Our findings are discussed toward designing robots that can take advantage of parental teaching.

Interesting objects are visually salient

How do we decide which objects in a visual scene are more interesting? While intuition may point toward high-level object recognition and cognitive processes, here we investigate the contributions of a much simpler process, low-level visual saliency. We used the LabelMe database (24,863 photographs with 74,454 manually outlined objects) to evaluate how often interesting objects were among the few most salient locations predicted by a computational model of bottom-up attention. In 43% of all images the model's predicted most salient location falls within a labeled region (chance 21%). Furthermore, in 76% of the images (chance 43%), one or more of the top three salient locations fell on an outlined object, with performance leveling off after six predicted locations. The bottom-up attention model has neither notion of object nor notion of semantic relevance. Hence, our results indicate that selecting interesting objects in a scene is largely constrained by low-level visual properties rather than solely determined by higher cognitive processes.

Dynamic visual selective attention model

We propose a new biologically motivated dynamic bottom-up selective attention model, which can generate a saliency map (SM) by considering dynamics of continuous input scenes as well as saliency of the primitive features of a static input scene. The maximum entropy algorithm is used to develop the dynamic selective attention model, whereby the input consists of the static bottom-up SMs for the successive static scenes. The experimental results show that the proposed model can generate more plausible scan paths for a dynamic scene compared with those obtained by the static bottom-up attention model.

Quantitative modelling of perceptual salience at human eye position

We investigate the extent to which a simple model of bottom-up attention and salience may be embedded within a broader computational framework, and compared with human eye movement data. We focus on quantifying whether increased simulation realism significantly affects quantitative measures of how well the model may predict where in video clips humans direct their gaze. We hence compare three variants of the model, tested with 15 video clips of natural scenes shown to three observers. We measure model-predicted salience at the locations gazed to by the observers, compared to random locations. The first variant simply processes the raw video clips. The second adds a gaze-contingent foveation filter. The third further attempts to realistically simulate dynamic human vision by embedding the video frames within a larger background, and shifting them to eye position. Our main finding is that increasing simulation realism significantly improves the predictive ability of the model. Better emulating the details of how a visual stimulus is captured by a constantly rotating retina during active vision has a significant positive impact onto quantitative comparisons between model and human behaviour.

The use of attention and spatial information for rapid facial recognition in video

Bottom-up visual attention allows primates to quickly select regions of an image that contain salient objects. In artificial systems, restricting the task of object recognition to these regions allows faster recognition and unsupervised learning of multiple objects in cluttered scenes. A problem with this approach is that objects superficially dissimilar to the target are given the same consideration in recognition as similar objects. In video, objects recognized in previous frames at locations distant to the current fixation point are given the same consideration in recognition as objects previously recognized in locations closer to the current target of attention. Due to the continuity of smooth motion, objects recently recognized in previous frames at locations close to the current focus of attention have a high probability of matching the current target. Here we investigate rapid pruning of the facial recognition search space using the already-computed low-level features that guide attention and spatial information derived from previous video frames. For each video frame, Itti & Koch's bottom-up visual attention algorithm is used to select salient locations based on low-level features such as contrast, orientation, color, intensity, flicker and motion. This algorithm has shown to be highly effective in selecting faces as salient objects. Lowe's SIFT object recognition algorithm then extracts a signature of the attended object, for comparison with the facial database. The database search is prioritized for faces which better match the low-level features used to guide attention to the current candidate for recognition or those that were previously recognized near the current candidate's location. The SIFT signatures of the prioritized faces are then checked against the attended candidate for a match. By comparing performance of Lowe's recognition algorithm and Itti & Koch's bottom-up attention model with or without search space pruning we demonstrate that our pruning approach improves the speed of facial recognition in video footage.