Scene Memorization Research Articles

Look at what I can do: Object affordances guide visual attention while speakers describe potential actions.

As we act on the world around us, our eyes seek out objects we plan to interact with. A growing body of evidence suggests that overt visual attention selects objects in the environment that could be interacted with, even when the task precludes physical interaction. In previous work, objects that afford grasping interactions influenced attention when static scenes depicted reachable spaces, and attention was otherwise better explained by general informativeness. Because grasping is but one of many object interactions, previous work may have downplayed the influence of object affordances on attention. The current study investigated the relationship between overt visual attention and object affordances versus broadly construed semantic information in scenes as speakers describe or memorize scenes. In addition to meaning and grasp maps-which capture informativeness and grasping object affordances in scenes, respectively-we introduce interact maps, which capture affordances more broadly. In a mixed-effects analysis of 5 eyetracking experiments, we found that meaning predicted fixated locations in a general description task and during scene memorization. Grasp maps marginally predicted fixated locations during action description for scenes that depicted reachable spaces only. Interact maps predicted fixated regions in description experiments alone. Our findings suggest observers allocate attention to scene regions that could be readily interacted with when talking about the scene, while general informativeness preferentially guides attention when the task does not encourage careful consideration of objects in the scene. The current study suggests that the influence of object affordances on visual attention in scenes is mediated by task demands.

Potsdam Eye-Movement Corpus for Scene Memorization and Search With Color and Spatial-Frequency Filtering.

Potsdam Eye-Movement Corpus for Scene Memorization and Search With Color and Spatial-Frequency Filtering.

Scene semantics outperform center bias during scene memorization, image saliency models do not

Scene semantics outperform center bias during scene memorization, image saliency models do not

Improving free-viewing fixation-related EEG potentials with continuous-time regression

BackgroundIn the analysis of combined ET-EEG data, there are several issues with estimating FRPs by averaging. Neural responses associated with fixations will likely overlap with one another in the EEG recording and neural responses change as a function of eye movement characteristics. Especially in tasks that do not constrain eye movements in any way, these issues can become confounds. New methodHere, we propose the use of regression based estimates as an alternative to averaging. Multiple regression can disentangle different influences on the EEG and correct for overlap. It thereby accounts for potential confounds in a way that averaging cannot. Specifically, we test the applicability of the rERP framework, as proposed by Smith and Kutas (2015b), (2017), or Sassenhagen (2018) to combined eye tracking and EEG data from a visual search and a scene memorization task. ResultsResults show that the method successfully estimates eye movement related confounds in real experimental data, so that these potential confounds can be accounted for when estimating experimental effects. Comparison with existing methodsThe rERP method successfully corrects for overlapping neural responses in instances where averaging does not. As a consequence, baselining can be applied without risking distortions. By estimating a known experimental effect, we show that rERPs provide an estimate with less variance and more accuracy than averaged FRPs. The method therefore provides a practically feasible and favorable alternative to averaging. ConclusionsWe conclude that regression based ERPs provide novel opportunities for estimating fixation related EEG in free-viewing experiments.

Task-Related Differences in Eye Movements in Individuals With Aphasia.

Background: Neurotypical young adults show task-based modulation and stability of their eye movements across tasks. This study aimed to determine whether persons with aphasia (PWA) modulate their eye movements and show stability across tasks similarly to control participants.Methods: Forty-eight PWA and age-matched control participants completed four eye-tracking tasks: scene search, scene memorization, text-reading, and pseudo-reading.Results: Main effects of task emerged for mean fixation duration, saccade amplitude, and standard deviations of each, demonstrating task-based modulation of eye movements. Group by task interactions indicated that PWA produced shorter fixations relative to controls. This effect was most pronounced for scene memorization and for individuals who recently suffered a stroke. PWA produced longer fixations, shorter saccades, and less variable eye movements in reading tasks compared to controls. Three-way interactions of group, aphasia subtype, and task also emerged. Text-reading and scene memorization were particularly effective at distinguishing aphasia subtype. Persons with anomic aphasia showed a reduction in reading saccade amplitudes relative to their respective control group and other PWA. Persons with conduction/Wernicke’s aphasia produced shorter scene memorization fixations relative to controls or PWA of other subtypes, suggesting a memorization specific effect. Positive correlations across most tasks emerged for fixation duration and did not significantly differ between controls and PWA.Conclusion: PWA generally produced shorter fixations and smaller saccades relative to controls particularly in scene memorization and text-reading, respectively. The effect was most pronounced recently after a stroke. Selectively in reading tasks, PWA produced longer fixations and shorter saccades relative to controls, consistent with reading difficulty. PWA showed task-based modulation of eye movements, though the pattern of results was somewhat abnormal relative to controls. All subtypes of PWA also demonstrated task-based modulation of eye movements. However, persons with anomic aphasia showed reduced modulation of saccade amplitude and smaller reading saccades, possibly to improve reading comprehension. Controls and PWA generally produced stabile fixation durations across tasks and did not differ in their relationship across tasks. Overall, these results suggest there is potential to differentiate among PWA with varying subtypes and from controls using eye movement measures of task-based modulation, especially reading and scene memorization tasks.

Meaning guides attention during scene viewing, even when it is irrelevant.

During real-world scene viewing, humans must prioritize scene regions for attention. What are the roles of low-level image salience and high-level semantic meaning in attentional prioritization? A previous study suggested that when salience and meaning are directly contrasted in scene memorization and preference tasks, attentional priority is assigned by meaning (Henderson & Hayes in Nature Human Behavior, 1, 743-747, 2017). Here we examined the role of meaning in attentional guidance using two tasks in which meaning was irrelevant and salience was relevant: a brightness rating task and a brightness search task. Meaning was represented by meaning maps that captured the spatial distribution of semantic features. Meaning was contrasted with image salience, represented by saliency maps. Critically, both maps were represented similarly, allowing us to directly compare how meaning and salience influenced the spatial distribution of attention, as measured by fixation density maps. Our findings suggest that even in tasks for which meaning is irrelevant and salience is relevant, meaningful scene regions are prioritized for attention over salient scene regions. These results support theories in which scene semantics play a dominant role in attentional guidance in scenes.

Observers' cognitive states modulate how visual inputs relate to gaze control.

Previous research has shown that eye-movements change depending on both the visual features of our environment, and the viewer's top-down knowledge. One important question that is unclear is the degree to which the visual goals of the viewer modulate how visual features of scenes guide eye-movements. Here, we propose a systematic framework to investigate this question. In our study, participants performed 3 different visual tasks on 135 scenes: search, memorization, and aesthetic judgment, while their eye-movements were tracked. Canonical correlation analyses showed that eye-movements were reliably more related to low-level visual features at fixations during the visual search task compared to the aesthetic judgment and scene memorization tasks. Different visual features also had different relevance to eye-movements between tasks. This modulation of the relationship between visual features and eye-movements by task was also demonstrated with classification analyses, where classifiers were trained to predict the viewing task based on eye movements and visual features at fixations. Feature loadings showed that the visual features at fixations could signal task differences independent of temporal and spatial properties of eye-movements. When classifying across participants, edge density and saliency at fixations were as important as eye-movements in the successful prediction of task, with entropy and hue also being significant, but with smaller effect sizes. When classifying within participants, brightness and saturation were also significant contributors. Canonical correlation and classification results, together with a test of moderation versus mediation, suggest that the cognitive state of the observer moderates the relationship between stimulus-driven visual features and eye-movements. (PsycINFO Database Record

Classifying mental states from eye movements during scene viewing.

How eye movements reflect underlying cognitive processes during scene viewing has been a topic of considerable theoretical interest. In this study, we used eye-movement features and their distributions over time to successfully classify mental states as indexed by the behavioral task performed by participants. We recorded eye movements from 72 participants performing 3 scene-viewing tasks: visual search, scene memorization, and aesthetic preference. To classify these tasks, we used statistical features (mean, standard deviation, and skewness) of fixation durations and saccade amplitudes, as well as the total number of fixations. The same set of visual stimuli was used in all tasks to exclude the possibility that different salient scene features influenced eye movements across tasks. All of the tested classification algorithms were successful in predicting the task within a single participant. The linear discriminant algorithm was also successful in predicting the task for each participant when the training data came from other participants, suggesting some generalizability across participants. The number of fixations contributed most to task classification; however, the remaining features and, in particular, their covariance provided important task-specific information. These results provide evidence on how participants perform different visual tasks. In the visual search task, for example, participants exhibited more variance and skewness in fixation durations and saccade amplitudes, but also showed heightened correlation between fixation durations and the variance in fixation durations. In summary, these results point to the possibility that eye-movement features and their distributional properties can be used to classify mental states both within and across individuals.

Dissociating temporal inhibition of return and saccadic momentum across multiple eye-movement tasks.

Saccade latencies are longer prior to an eye movement to a recently fixated location than to control locations, a phenomenon known as oculomotor inhibition of return (O-IOR). There are theoretical reasons to expect that O-IOR would vary in magnitude across different eye movement tasks, but previous studies have produced contradictory evidence. However, this may have been because previous studies have not dissociated O-IOR and a related phenomenon, saccadic momentum, which is a bias to repeat saccade programs that also influences saccade latencies. The present study dissociated the influence of O-IOR and saccadic momentum across three complex visual tasks: scene search, scene memorization, and scene aesthetic preference. O-IOR was of similar magnitude across all three tasks, while saccadic momentum was weaker in scene search.

Dissociating temporal inhibition of return and saccadic momentum across multiple eye-movement tasks

Saccade latencies are longer before eye movements to recently fixated locations, a phenomenon known as oculomotor inhibition of return (O-IOR). However, latencies are also influenced by saccadic momentum: latencies are shortest before forward saccades and increase as the angular deviation (difference in direction) of the current and previous saccades increases. Using a within-subjects and within-items design, the present study attempted to dissociate the temporal and spatial consequences of O-IOR and saccadic momentum across three tasks: scene search, scene memorization, and aesthetic preference. Spatial analyses revealed facilitation of return (rather than inhibition) in all tasks: return saccades were more frequent in the data compared to a shuffled baseline. Initial temporal analyses suggested that O-IOR was weaker in search (16 ms) than in memorization or preference (28 & 25 ms, respectively). Saccadic momentum appears even if previous and current saccades are not closely matched in amplitude, while O-IOR appears only when saccades are matched in amplitude so the current saccade lands within the zone of IOR (within 4° of the previous fixation location), making it possible to dissociate the two phenomena. Fixation durations increased linearly as a function of angular deviation outside the IOR zone, but curvilinearly within the zone of IOR, revealing an additional penalty for return saccades beyond that accounted for by saccadic momentum. Furthermore, saccadic momentum and O-IOR differentially affected the frequency distribution of fixation durations. Saccadic momentum shifted the mean of the distribution's Gaussian component, lengthening most fixations, while O-IOR only influenced the skewness of the distribution (the exponential component), lengthening a subset of fixations. Interestingly, when dissociated in this way, O-IOR was equivalent in magnitude across tasks while saccadic momentum was not present in search. These results suggest that O-IOR and saccadic momentum are independent phenomena, and that O-IOR is task-independent while saccadic momentum is not. Meeting abstract presented at VSS 2014

Task dependent memory recall performance of naturalistic scenes: Incidental memorization during search outperforms intentional scene memorization

Memorizing critical objects and their locations is an essential part of everyday life. In the present study, incidental encoding of objects in scenes during search was compared to explicit memorization of those scenes. Participants were shown 10 different, indoor scenes. 150 objects (15/scene) were preselected. Regions of interest were defined around each object for eye tracking analysis. Observers searched for ten objects in each of five scenes (50 trials, mean RT=3550ms). For the other five scenes, observers spent 15 seconds memorizing as much as possible of each scene for later recall (task order and scenes assignments counterbalanced). We subsequently tested explicit scene memory in two ways. Global scene representation / boundary extension was assessed by showing observers previously-presented scenes and asking if the scene was closer-up or further away than the “original”. Detailed scene memory was assessed by asking participants to redraw each of the 10 scenes and their objects. We found no indication of boundary extension for these complex, indoor scenes. Inferences about the scene beyond the image border did not become part of scene memory. Overall, a rather small percentage of the objects were subsequently drawn. Interestingly, even though participants in the search condition were not explicitly asked to memorize the scenes, they reproduced a substantially greater number of objects (22%) compared to the memory condition (11%). This advantage was produced by 29% recall of search targets, which received highest gaze durations (2600ms). Only 9% of all distractors were recalled despite mean gaze durations of 1730ms. Objects in the memorize condition were only looked at for 700ms, but 11% of these objects were recalled suggesting differential, task-dependent encoding strategies. Instructions to search for specific objects produced stronger encoding than the general request to memorize the scene even though the critical objects were repeatedly fixated in both conditions.

Statistics of eye movements in scene categorization and scene memorization

Humans can grasp the gist of complex natural scenes in less than 100 milliseconds. To model this rapid scene perception, we developed ultra-sparse structural representations of natural scenes. In this model, a natural scene is represented by a probability distribution based on a set of natural scene structures and their spatial concatenations. In contrast to bottom-up, image-based processing and recent scene gist models based on global features, the structural representations in our model require no isolation of objects, nor computation of global scene features. We tested this model by comparing the patterns of eye movements of human subjects while they were performing two scene-viewing tasks, scene categorization and scene memorization[1,2]. Since human vision system always actively seeks visual information needed to support current cognitive requirements[3], we predicted that the subjects would scan more informative patches of the scenes in scene categorization than in scene memorization. We examined the statistics of the eye movements of the subjects in the two tasks. We found that: 1) the average number of fixations in scene categorization was significantly less than that in scene memorization (t = 7.02, p < 0.001); 2) the average fixation duration in scene categorization was significant longer than that in scene memorization (t = 2, p = 0.05); 3) the subjects executed more long fixations in scene categorization than in scene memorization; and 4) the total length of the scan paths in scene categorization was shorter than that in scene memorization (t = 5.48, p < 0.01). We also developed models of scene categorization and scene memorization using the scene patches sampled at the fixations and found that the subjects extracted more informative scene patches in scene categorization than in scene memorization. Therefore, the subjects tended to coarsely scan the whole scenes when performing scene memorization, but opted to scan smaller scene patches with more detailed information when performing scene categorization. This observation supports our model of visual scenes as spatial concatenations of natural scene structures, each of which conveys a variety of amount of information about scene identity and category.

Facilitation of return during scene viewing

Inhibition of Return (IOR) is a delay in initiating attentional shifts to previously attended locations. It is believed to facilitate attentional exploration of a scene. Computational models of attention have implemented IOR as a simple mechanism for driving attention through a scene. However, evidence for IOR during scene viewing is inconclusive. In this study IOR during scene memorization and in response to sudden onsets at the last (1-back) and penultimate (2-back) fixation location was measured. The results indicate that there is a tendency for saccades to continue the trajectory of the last saccade (Saccadic Momentum), but contrary to the “foraging facilitator” hypothesis of IOR, there is also a distinct population of saccades directed back to the last fixation location, especially in response to onsets. Voluntary return saccades to the 1-back location experience temporal delay but this does not affect their likelihood of occurrence. No localized temporal delay is exhibited at 2-back. These results suggest that IOR exists at the last fixation location during scene memorization but that this temporal delay is overridden by Facilitation of Return. Computational models of attention will fail to capture the pattern of saccadic eye movements during scene viewing unless they model the dynamics of visual encoding and can account for the interaction between Facilitation of Return, Saccadic Momentum, and Inhibition of Return.

How are eye fixation durations controlled during scene viewing? Further evidence from a scene onset delay paradigm

Recent research on eye movements during scene viewing has focused on where the eyes fixate. But eye fixations also differ in their durations. Here we investigated whether fixation durations in scene viewing are under the direct and immediate control of the current visual input. In two scene memorization and one visual search experiments, the scene was removed from view during critical fixations for a predetermined delay, and then restored following the delay. Experiment 1 compared filled (pattern mask) and unfilled (grey field) delays. Experiment 2 compared random to blocked delays. Experiment 3 extended the results to a visual search task. The results demonstrate that fixation durations in scene viewing comprise two fixation populations. One population remains relatively constant across delay, and the second population increases with scene onset delay. The results are consistent with a mixed eye movement control model that incorporates an autonomous control mechanism with process monitoring. The results suggest that a complete gaze control model will have to account for both fixation location and fixation duration.

Incidental visual memory for objects in scenes

Three experiments were conducted to investigate the existence of incidentally acquired, long-term, detailed visual memory for objects embedded in previously viewed scenes. Participants performed intentional memorization and incidental visual search learning tasks while viewing photographs of real-world scenes. A visual memory test for previously viewed objects from these scenes then followed. Participants were not aware that they would be tested on the scenes following incidental learning in the visual search task. In two types of memory tests for visually specific object information (token discrimination and mirror-image discrimination), performance following both the memorization and visual search conditions was reliably above chance. These results indicate that recent demonstrations of good visual memory during scene viewing are not due to intentional scene memorization. Instead, long-term visual representations are incidentally generated as a natural product of scene perception.