Overt Visual Attention Research Articles

Perceived interest and overt visual attention in natural images

Region of interest (ROI) based image and video processing has attracted increased research efforts in recent years. The concept of perceptual ROI, however, is not always clearly defined leading to different interpretations between researchers related to bottom-up saliency (signal driven visual attention), top-down attention (subconscious, driven by higher cognitive factors, e.g. interest) or perceived interest. While all of these concepts are likely meaningful in the context of perceptual ROI based image and video processing, it is worth understanding how they are linked altogether. In this paper, the relationship between perceived interest and overt visual attention (which can cover both bottom-up and top-down attention) is studied. Towards this goal, a dedicated ROI selection experiment was performed and is analysed in detail, revealing deep insight into perceived interest in natural images. The outcomes are compared to an eye gaze tracking experiment representing overt visual attention in natural images. It is shown that there is indeed a strong relationship between perceived interest and overt visual attention for a wide range of natural scenes. We show that the relationship has a strong dependence on image content and presentation time during the eye gaze tracking experiment. Furthermore, eye gaze tracking data is revealed to have a high predictive value of primary ROI, particularly in case of the latter dominating over the remainder of the image. Both, the ROI and the eye gaze tracking databases are made publicly available to the research community.

Effects of contextual information and stimulus ambiguity on overt visual sampling behavior

The sampling of our visual environment through saccadic eye movements is an essential function of the brain, allowing us to overcome the limits of peripheral vision. Understanding which parts of a scene attract overt visual attention is subject to intense research, and considerable progress has been made in unraveling the underlying cortical mechanisms. In contrast to spatial aspects, however, relatively little is understood about temporal aspects of overt visual sampling. At every fixation, the oculomotor system faces the decision whether to keep exploring different aspects of an object or scene or whether to remain fixated to allow for in-depth cortical processing – a situation that can be understood in terms of an exploration–exploitation dilemma. To improve our understanding of the factors involved in these decisions, we here investigate how the level of visual information, experimentally manipulated by scene context and stimulus ambiguity, changes the sampling behavior preceding the recognition of centrally presented ambiguous and disambiguated objects. Behaviorally, we find that context, although only presented until the first voluntary saccade, biases the perceptual outcome and significantly reduces reaction times. Importantly, we find that increased information about an object significantly alters its visual exploration, as evident through increased fixation durations and reduced saccade amplitudes. These results demonstrate that the initial sampling of an object, preceding its recognition, is subject to change based on the amount of information available in the system: increased evidence for its identity biases the exploration–exploitation strategy towards in-depth analyses.

The role of the right frontal eye field in overt visual attention deployment as assessed by free visual exploration

The frontal eye field (FEF) is known to be involved in saccade generation and visual attention control. Studies applying covert attentional orienting paradigms have shown that the right FEF is involved in attentional shifts to both the left and the right hemifield. In the current study, we aimed at examining the effects of inhibitory continuous theta burst (cTBS) transcranial magnetic stimulation over the right FEF on overt attentional orienting, as measured by a free visual exploration paradigm.In forty-two healthy subjects, free visual exploration of naturalistic pictures was tested in three conditions: (1) after cTBS over the right FEF; (2) after cTBS over a control site (vertex); and, (3) without any stimulation. The results showed that cTBS over the right FEF–but not cTBS over the vertex–triggered significant changes in the spatial distribution of the cumulative fixation duration. Compared to the group without stimulation and the group with cTBS over the vertex, cTBS over the right FEF decreased cumulative fixation duration in the left and in the right peripheral regions, and increased cumulative fixation duration in the central region.The present study supports the view that the right FEF is involved in the bilateral control of not only covert, but also of overt, peripheral visual attention.

Haptic guidance of overt visual attention.

Research has shown that information accessed from one sensory modality can influence perceptual and attentional processes in another modality. Here, we demonstrated a novel crossmodal influence of haptic-shape information on visual attention. Participants visually searched for a target object (e.g., an orange) presented among distractor objects, fixating the target as quickly as possible. While searching for the target, participants held (never viewed and out of sight) an item of a specific shape in their hands. In two experiments, we demonstrated that the time for the eyes to reach a target-a measure of overt visual attention-was reduced when the shape of the held item (e.g., a sphere) was consistent with the shape of the visual target (e.g., an orange), relative to when the held shape was unrelated to the target (e.g., a hockey puck) or when no shape was held. This haptic-to-visual facilitation occurred despite the fact that the held shapes were not predictive of the visual targets' shapes, suggesting that the crossmodal influence occurred automatically, reflecting shape-specific haptic guidance of overt visual attention.

What/Where to Look Next? Modeling Top-Down Visual Attention in Complex Interactive Environments

Several visual attention models have been proposed for describing eye movements over simple stimuli and tasks such as free viewing or visual search. Yet, to date, there exists no computational framework that can reliably mimic human gaze behavior in more complex environments and tasks such as urban driving. In addition, benchmark datasets, scoring techniques, and top-down model architectures are not yet well understood. In this paper, we describe new task-dependent approaches for modeling top-down overt visual attention based on graphical models for probabilistic inference and reasoning. We describe a dynamic Bayesian network that infers probability distributions over attended objects and spatial locations directly from observed data. Probabilistic inference in our model is performed over object-related functions that are fed from manual annotations of objects in video scenes or by state-of-the-art object detection/recognition algorithms. Evaluating over approximately 3 h (approximately 315 000 eye fixations and 12 000 saccades) of observers playing three video games (time-scheduling, driving, and flight combat), we show that our approach is significantly more predictive of eye fixations compared to: 1) simpler classifier-based models also developed here that map a signature of a scene (multimodal information from gist, bottom-up saliency, physical actions, and events) to eye positions; 2) 14 state-of-the-art bottom-up saliency models; and 3) brute-force algorithms such as mean eye position. Our results show that the proposed model is more effective in employing and reasoning over spatio-temporal visual data compared with the state-of-the-art.

Toward a Redefinition of “Educational” in Babies’ and Toddlers’ Media:Overt Visual Attention, Verbalization, and Other Measures of Engagement as Indicators of Learning

This paper provides an analysis of data from an earlier study in which children 12- to 24-months failed to learn from an educational infant DVD. Whereas children performed no better than chance on the original posttests, coders noticed verbal and nonverbal behaviors that appeared to indicate engagement with the content while it was playing. After an analysis of the videotaped trials, this project uncovered several behaviors indicating engagement, such as unprompted verbalizations of items on screen, verbal imitations of items on screen, non-verbal indications of engagement with the program, program elicited amusement, and both verbal and nonverbal behavior directed at the DVD player itself. Results are discussed in the context of Sproull (1973) and Crawley et al.’s (2002) findings that direct reactions to program content are indicators of “vicarious participation” which is a reflection, rather than a cause, of learning.

Time-course of attention biases in social phobia

Theoretical models of social phobia implicate preferential attention to social threat in the maintenance of anxiety symptoms, though there has been limited work characterizing the nature of these biases over time. The current study utilized eye-movement data to examine the time-course of visual attention over 1500ms trials of a probe detection task. Nineteen participants with a primary diagnosis of social phobia based on DSM-IV criteria and 20 non-clinical controls completed this task with angry, fearful, and happy face trials. Overt visual attention to the emotional and neutral faces was measured in 50ms segments across the trial. Over time, participants with social phobia attend less to emotional faces and specifically less to happy faces compared to controls. Further, attention to emotional relative to neutral expressions did not vary notably by emotion for participants with social phobia, but control participants showed a pattern after 1000ms in which over time they preferentially attended to happy expressions and avoided negative expressions. Findings highlight the importance of considering attention biases to positive stimuli as well as the pattern of attention between groups. These results suggest that attention “bias” in social phobia may be driven by a relative lack of the biases seen in non-anxious participants.

Effects of uni- and multimodal cueing on handrail grasping and associated gaze behavior in older adults

IntroductionIt appears that age-related changes in visual attention may impair ability to acquire the visuospatial information needed to grasp a handrail effectively in response to sudden loss of balance. This, in turn, may increase risk of falling. To counter this problem, we developed a proximity-triggered cueing system that provides a visual cue (flashing lights) and/or verbal cue (“attention use the handrail”) to attract attention to the handrail. This study examined the effect of handrail cueing on grasping of the rail and associated gaze behavior in a large cohort (n=160) of independent and ambulatory older adults (age 64–80). MethodsThe handrail and cueing system was mounted on a large (2m×6m) motion platform configured to simulate a real-life environment. Subjects performed a daily-life task that required walking to the end of the platform, which was triggered to perturb balance by moving suddenly when they were adjacent to the rail. To prevent adaptation, each subject performed only one trial, and a deception was used to ensure that the perturbation was truly unexpected. Each subject was assigned to one of four cue conditions: visual, verbal, multimodal (visual-plus-verbal) or no cue. ResultsVerbal cueing attracted overt visual attention to the handrail and markedly increased proactive grasping (prior to the onset of the balance perturbation) particularly when delivered unimodally. Subjects were otherwise much more likely to grasp the rail in reaction to the perturbation. A possible trend for visual cueing to improve the accuracy of these reactions was offset by adverse effects on reaction speed and on frequency of proactive grasping. ConclusionsThe results support the viability of using unimodal verbal cueing to reduce fall risk by increasing proactive handrail use. Conversely, they do not strongly support use of visual cueing (either alone or in combination with verbal cueing) and suggest that it may even have adverse effects. Further study is needed to evaluate effects of handrail cueing in a wide range of populations and real-life settings.

Identifying the Computational Requirements of an Integrated Top-Down-Bottom-Up Model for Overt Visual Attention within an Active Vision System

Computational visual attention systems have been constructed in order for robots and other devices to detect and locate regions of interest in their visual world. Such systems often attempt to take account of what is known of the human visual system and employ concepts, such as ‘active vision’, to gain various perceived advantages. However, despite the potential for gaining insights from such experiments, the computational requirements for visual attention processing are often not clearly presented from a biological perspective. This was the primary objective of this study, attained through two specific phases of investigation: 1) conceptual modeling of a top-down-bottom-up framework through critical analysis of the psychophysical and neurophysiological literature, 2) implementation and validation of the model into robotic hardware (as a representative of an active vision system). Seven computational requirements were identified: 1) transformation of retinotopic to egocentric mappings, 2) spatial memory for the purposes of medium-term inhibition of return, 3) synchronization of ‘where’ and ‘what’ information from the two visual streams, 4) convergence of top-down and bottom-up information to a centralized point of information processing, 5) a threshold function to elicit saccade action, 6) a function to represent task relevance as a ratio of excitation and inhibition, and 7) derivation of excitation and inhibition values from object-associated feature classes. The model provides further insight into the nature of data representation and transfer between brain regions associated with the vertebrate ‘active’ visual attention system. In particular, the model lends strong support to the functional role of the lateral intraparietal region of the brain as a primary area of information consolidation that directs putative action through the use of a ‘priority map’.

Intranasal oxytocin increases covert attention to positive social cues

The neuropeptide oxytocin (OT) has positive effects on the processing of emotional stimuli such as facial expressions. To date, research has focused primarily on conditions of overt visual attention. We investigated whether a single intranasal dose of OT (24 IU) would modulate the allocation of attentional resources towards positive and negative facial expressions using a dot-probe paradigm in a sample of 69 healthy men. Attentional capacity for these facial cues was limited by presentation time (100 or 500 ms). In addition, we controlled for overt visual attention by recording eye movements using a remote eye tracker. Reaction times (RTs) in the dot-probe paradigm revealed a pronounced shift of attention towards happy facial expressions presented for 100 ms after OT administration, whereas there were no OT-induced effects for longer presentation times (500 ms). The results could not be attributed to modulations of overt visual attention as no substance effects on gazes towards the facial target were observed. The results suggest that OT increased covert attention to happy faces, thereby supporting the hypothesis that OT modulates early attentional processes that might promote prosocial behavior.

Unmasking the contribution of low-level features to the guidance of attention

The role of low-level stimulus-driven control in the guidance of overt visual attention has been difficult to establish because low- and high-level visual content are spatially correlated within natural visual stimuli. Here we show that impairment of parietal cortical areas, either permanently by a lesion or reversibly by repetitive transcranial magnetic stimulation (rTMS), leads to fixation of locations with higher values of low-level features as compared to control subjects or in a no-rTMS condition. Moreover, this unmasking of stimulus-driven control crucially depends on the intrahemispheric balance between top-down and bottom-up cortical areas. This result suggests that although in normal behavior high-level features might exert a strong influence, low-level features do contribute to guide visual selection during the exploration of complex natural stimuli.

The emotion-induced memory trade-off: More than an effect of overt attention?

Although it has been suggested that many effects of emotion on memory are attributable to attention, in the present study we addressed the hypothesis that such effects may relate to a number of different factors during encoding or postencoding. One way to look at the effects of emotion on memory is by examining the emotion-induced memory trade-off, whereby enhanced memory for emotional items often comes at the cost of memory for surrounding background information. We present evidence that this trade-off cannot be explained solely by overt attention (measured via eyetracking) directed to the emotional items during encoding. Participants did not devote more overt attention to emotional than to neutral items when those items were selectively remembered (at the expense of their backgrounds). Only when participants were asked to answer true/false questions about the items and the backgrounds--a manipulation designed to affect both overt attention and poststimulus elaboration--was there a reduction in selective emotional item memory due to an increase in background memory. These results indicate that the allocation of overt visual attention during encoding is not sufficient to predict the occurrence of selective item memory for emotional items.

Visual and memory search in complex environments: determinants of eye movements and search performance

Previous research on visual and memory search revealed various top down and bottom up factors influencing performance. However, utilising abstract stimuli (e.g. geometrical shapes or letters) and focussing on individual factors has often limited the applicability of research findings. Two experiments were designed to analyse which attributes of a product facilitate search in an applied environment. Participants scanned displays containing juice packages while their eye movements were recorded. The familiarity, saliency, and position of search targets were systematically varied. Experiment 1 involved a visual search task, whereas Experiment 2 focussed on memory search. The results showed that bottom up (target saliency) and top down (target familiarity) factors strongly interacted. Overt visual attention was influenced by cultural habits, purposes, and current task demands. The results provide a solid database for assessing the impact and interplay of fundamental top down and bottom up determinants of search processes in applied fields of psychology. Practitioner Summary: Our study demonstrates how a product (or a visual item in general) needs to be designed and placed to ensure that it can be found effectively and efficiently within complex environments. Corresponding product design should result in faster and more accurate visual and memory based search processes.

High-resolution Functional Magnetic Resonance Imaging Methods for Human Midbrain

Functional MRI (fMRI) is a widely used tool for non-invasively measuring correlates of human brain activity. However, its use has mostly been focused upon measuring activity on the surface of cerebral cortex rather than in subcortical regions such as midbrain and brainstem. Subcortical fMRI must overcome two challenges: spatial resolution and physiological noise. Here we describe an optimized set of techniques developed to perform high-resolution fMRI in human SC, a structure on the dorsal surface of the midbrain; the methods can also be used to image other brainstem and subcortical structures. High-resolution (1.2 mm voxels) fMRI of the SC requires a non-conventional approach. The desired spatial sampling is obtained using a multi-shot (interleaved) spiral acquisition (1). Since, T(2)* of SC tissue is longer than in cortex, a correspondingly longer echo time (T(E) ~ 40 msec) is used to maximize functional contrast. To cover the full extent of the SC, 8-10 slices are obtained. For each session a structural anatomy with the same slice prescription as the fMRI is also obtained, which is used to align the functional data to a high-resolution reference volume. In a separate session, for each subject, we create a high-resolution (0.7 mm sampling) reference volume using a T(1)-weighted sequence that gives good tissue contrast. In the reference volume, the midbrain region is segmented using the ITK-SNAP software application (2). This segmentation is used to create a 3D surface representation of the midbrain that is both smooth and accurate (3). The surface vertices and normals are used to create a map of depth from the midbrain surface within the tissue (4). Functional data is transformed into the coordinate system of the segmented reference volume. Depth associations of the voxels enable the averaging of fMRI time series data within specified depth ranges to improve signal quality. Data is rendered on the 3D surface for visualization. In our lab we use this technique for measuring topographic maps of visual stimulation and covert and overt visual attention within the SC (1). As an example, we demonstrate the topographic representation of polar angle to visual stimulation in SC.

High-resolution Functional Magnetic Resonance Imaging Methods for Human Midbrain

Functional MRI (fMRI) is a widely used tool for non-invasively measuring correlates of human brain activity. However, its use has mostly been focused upon measuring activity on the surface of cerebral cortex rather than in subcortical regions such as midbrain and brainstem. Subcortical fMRI must overcome two challenges: spatial resolution and physiological noise. Here we describe an optimized set of techniques developed to perform high-resolution fMRI in human SC, a structure on the dorsal surface of the midbrain; the methods can also be used to image other brainstem and subcortical structures. High-resolution (1.2 mm voxels) fMRI of the SC requires a non-conventional approach. The desired spatial sampling is obtained using a multi-shot (interleaved) spiral acquisition1. Since, T2* of SC tissue is longer than in cortex, a correspondingly longer echo time (TE ~ 40 msec) is used to maximize functional contrast. To cover the full extent of the SC, 8-10 slices are obtained. For each session a structural anatomy with the same slice prescription as the fMRI is also obtained, which is used to align the functional data to a high-resolution reference volume. In a separate session, for each subject, we create a high-resolution (0.7 mm sampling) reference volume using a T1-weighted sequence that gives good tissue contrast. In the reference volume, the midbrain region is segmented using the ITK-SNAP software application2. This segmentation is used to create a 3D surface representation of the midbrain that is both smooth and accurate3. The surface vertices and normals are used to create a map of depth from the midbrain surface within the tissue4. Functional data is transformed into the coordinate system of the segmented reference volume. Depth associations of the voxels enable the averaging of fMRI time series data within specified depth ranges to improve signal quality. Data is rendered on the 3D surface for visualization. In our lab we use this technique for measuring topographic maps of visual stimulation and covert and overt visual attention within the SC1. As an example, we demonstrate the topographic representation of polar angle to visual stimulation in SC.

Head and eye gaze dynamics during visual attention shifts in complex environments

The dynamics of overt visual attention shifts evoke certain patterns of responses in eye and head movements. In this work, we detail novel findings regarding the interaction of eye gaze and head pose under various attention-switching conditions in complex environments and safety critical tasks such as driving. In particular, we find that sudden, bottom-up visual cues in the periphery evoke a different pattern of eye-head movement latencies as opposed to those during top-down, task-oriented attention shifts. In laboratory vehicle simulator experiments, a unique and significant (p < 0.05) pattern of preparatory head motions, prior to the gaze saccade, emerges in the top-down case. This finding is validated in qualitative analysis of naturalistic real-world driving data. These results demonstrate that measurements of eye-head dynamics are useful data for detecting driver distractions, as well as in classifying human attentive states in time and safety critical tasks.

Decoding Successive Computational Stages of Saliency Processing

An important requirement for vision is to identify interesting and relevant regions of the environment for further processing. Some models assume that salient locations from a visual scene are encoded in a dedicated spatial saliency map [1, 2]. Then, a winner-take-all (WTA) mechanism [1, 2] is often believed to threshold the graded saliency representation and identify the most salient position in the visual field. Here we aimed to assess whether neural representations of graded saliency and the subsequent WTA mechanism can be dissociated. We presented images of natural scenes while subjects were in a scanner performing a demanding fixation task, and thus their attention was directed away. Signals in early visual cortex and posterior intraparietal sulcus (IPS) correlated with graded saliency as defined by a computational saliency model. Multivariate pattern classification [3, 4] revealed that the most salient position in the visual field was encoded in anterior IPS and frontal eye fields (FEF), thus reflecting a potential WTA stage. Our results thus confirm that graded saliency and WTA-thresholded saliency are encoded in distinct neural structures. This could provide the neural representation required for rapid and automatic orientation toward salient events in natural environments.

Intranasal oxytocin enhances emotion recognition from dynamic facial expressions and leaves eye-gaze unaffected

Previous studies have shown that oxytocin improves the encoding and recognition of facial expressions, which has been proposed to be mediated by an increased exploration of the eye region during face processing. In the present study, we used eye tracking to assess visual attention to the eye region while participants performed a dynamic facial emotion recognition task. In a double-blind, placebo-controlled between-subjects design participants received 24 IU intranasal oxytocin (n = 23) or a placebo (n = 24). Although oxytocin administration had no effect on participants' visual scanning of emotional faces, it generally enhanced recognition performance, as the oxytocin group recognized emotional expressions at lower intensity levels. These findings suggest that oxytocin-induced improvement of facial emotion recognition is independent of modulations in overt visual attention.

Overt Visual Attention as a Causal Factor of Perceptual Awareness

Our everyday conscious experience of the visual world is fundamentally shaped by the interaction of overt visual attention and object awareness. Although the principal impact of both components is undisputed, it is still unclear how they interact. Here we recorded eye-movements preceding and following conscious object recognition, collected during the free inspection of ambiguous and corresponding unambiguous stimuli. Using this paradigm, we demonstrate that fixations recorded prior to object awareness predict the later recognized object identity, and that subjects accumulate more evidence that is consistent with their later percept than for the alternative. The timing of reached awareness was verified by a reaction-time based correction method and also based on changes in pupil dilation. Control experiments, in which we manipulated the initial locus of visual attention, confirm a causal influence of overt attention on the subsequent result of object perception. The current study thus demonstrates that distinct patterns of overt attentional selection precede object awareness and thereby directly builds on recent electrophysiological findings suggesting two distinct neuronal mechanisms underlying the two phenomena. Our results emphasize the crucial importance of overt visual attention in the formation of our conscious experience of the visual world.

The influence of synesthesia on eye movements: No synesthetic pop-out in an oculomotor target selection task

Recent research on grapheme-colour synesthesia has focused on whether visual attention is necessary to induce a synesthetic percept. The current study investigated the influence of synesthesia on overt visual attention during an oculomotor target selection task. Chromatic and achromatic stimuli were presented with one target among distractors (e.g. a '2' (target) among multiple '5's (distractors)). Participants executed an eye movement to the target. Synesthetes and controls showed a comparable target selection performance across conditions and a 'pop-out effect' was only seen in the chromatic condition. As a pop-out effect was absent for the synesthetes in the achromatic condition, a synesthetic element appears not to elicit a synesthetic colour, even when it is the target. The synesthetic percepts are not pre-attentively available to distinguish the synesthetic target from synesthetic distractors when elements are presented in the periphery. Synesthesia appears to require full recognition to bind form and colour.