Rapid Scene Categorization Research Articles

A Foveated Vision-Transformer Model for Scene Classification

Introduction: Humans can rapidly categorize scenes (R. VanRullen & S. Thorpe, 2001), even using peripheral vision (Larson & Loschky, 2009). Various computational models have been proposed for rapid scene categorization in terms of low-level properties such as spatial envelopes (Oliva & Torralba, 2001) and texture summary statistics (TTM, Rosenholtz et al., 2012). Yet, these models do not explicitly model the foveated properties of the visual system nor the interaction between eye movements and the scene category task. We propose a model with a foveated visual system and eye movements that can predict the dependence of human categorization performance across fixations. The model combines square pooling regions with the computer vision-transformer architecture (Dosovitskiy et al., 2020, Touvron et al., 2020) and makes multiple fixations to maximize classification using the technique of self-attention (Parikh et al., 2016, Bahdanau et al., 2015). Methods: Twenty-two participants classified 360 images (Places365 database, places2.csail.mit.edu) into 30 classes. Images subtended a viewing angle of 22.7 degrees. A gaze-contingent display was used to randomly interrupt the display after 1, 2, 3, or 4 fixations with initial forced-fixation at bottom-center or top-center. Results: We show that there is no significant improvement in performance after the 2nd fixation (Δ correct categorization=0.015; p=0.4729), unlike performance for object search (Koehler and Eckstein, 2017). The model correctly predicts modest classification improvements for free-viewing fixations (Δ=0.016). The model-human correlation in classification choices was not significantly lower than human-human correlations. Our findings suggest that human categorization of scenes within a single fixation can be explained by the spatially global distribution of the visual information in the scene and their availability even through the bottlenecks of the visual periphery. The newly proposed hybrid approach using biologically based modeling and Transformers can flexibly be applied to various naturalistic tasks and stimuli.

Rapid scene categorization is not purely feed-forward: An EEG investigation of scene gist facilitation by sequential predictions

Rapid scene categorization is not purely feed-forward: An EEG investigation of scene gist facilitation by sequential predictions

The role of high-order statistics in evoking perceptual priming effect on rapid scene categorization

The role of high-order statistics in evoking perceptual priming effect on rapid scene categorization

Rapid scene categorization: From coarse peripheral vision to fine central vision

Studies on scene perception have shown that the rapid extraction of low spatial frequencies (LSF) allows a coarse parsing of the scene, prior to the analysis of high spatial frequencies (HSF) containing details. Many studies suggest that scene gist recognition can be achieved with only the low resolution of peripheral vision. Our study investigated the advantage of peripheral vision on central vision during a scene categorization task (indoor vs. outdoor). In Experiment 1, we used large scene photographs from which we built one central disk and four circular rings of different eccentricities. The central disk either contained or not an object semantically related to the scene category. Results showed better categorization performances for the peripheral rings, despite the presence of an object in central vision that was semantically related to the scene category that significantly improved categorization performances. In Experiment 2, the central disk and rings were assembled from Central to Peripheral vision (CtP sequence) or from Peripheral to Central vision (PtC sequence). Results revealed better performances for PtC than CtP sequences, except when no central object was present under rapid categorization constraints. As Experiment 3 suggested that the PtC advantage was not explained by a reduction of the visibility of the object in the central disk by the surrounding peripheral rings (CtP sequence), results are interpreted in the context of a predominant coarse-to-fine processing during scene categorization, with greater efficiency and utility of coarse peripheral vision relative to fine central vision during rapid scene categorization.

The contributions of central and peripheral vision to scene-gist recognition with a 180° visual field.

We investigated the relative contributions of central versus peripheral vision in scene-gist recognition with panoramic 180° scenes. Experiment 1 used the window/scotoma paradigm of Larson and Loschky (2009). We replicated their findings that peripheral vision was more important for rapid scene categorization, while central vision was more efficient, but those effects were greatly magnified. For example, in comparing our critical radius (which produced equivalent performance with mutually exclusive central and peripheral image regions) to that of Larson and Loschky, our critical radius of 10° had a ratio of central to peripheral image area that was 10 times smaller. Importantly, we found different functional relationships between the radius of centrally versus peripherally presented imagery (or the proportion of centrally versus peripherally presented image area) and scene-categorization sensitivity. For central vision, stimulus discriminability was an inverse function of image radius, while for peripheral vision the relationship was essentially linear. In Experiment 2, we tested the photographic-bias hypothesis that the greater efficiency of central vision for rapid scene categorization was due to more diagnostic information in the center of photographs. We factorially compared the effects of the eccentricity from which imagery was sampled versus the eccentricity at which imagery was presented. The presentation eccentricity effect was roughly 3 times greater than the sampling eccentricity effect, showing that the central-vision efficiency advantage was primarily due to the greater sensitivity of central vision. We discuss our results in terms of the eccentricity-dependent neurophysiology of vision and discuss implications for computationally modeling rapid scene categorization.

The effects of distributed and focused attention on rapid scene categorization

ABSTRACTIt has been argued that distributed attention facilitates the rapid extraction of summary statistics that underpins rapid scene categorization. We directly examined this hypothesis by investigating whether distributed or focused attention is more compatible with the extraction of both summary statistics (Experiment 1) and semantic scene information (Experiments 2–4). Experiment 1 replicated Chong and Treisman’s (2005a) result that mean circle size judgments are more compatible with a distributed attention task than a focused attention task. Experiment 2 investigated whether this finding extends to simple scene categorization by replacing the averaging task with an animal detection task. Consistent with Experiment 1, the ability to detect the presence of an animal was more compatible with a distributed attention task than a focused attention task. Experiments 3 and 4 addressed whether distributed attention influences scene categorization tasks. When observers were asked to classify scenes based on their basic level (e.g., beach or forest; Experiment 3), there was no statistically significant difference between focused and distributed attention task conditions; however, superordinate level categorization (e.g., natural or manmade; Experiment 4) was faster when combined with a task requiring distributed attention compared to a task requiring focused attention.

Narrative priming of scene gist: The role of sequential expectations in scene gist perception

Observers can categorize a novel scene within the first 100 ms of its onset. Researchers have suggested this is accomplished through a rapid feed-forward sweep of neural activation. Consequently, researchers have focused on examining the minimal perceptual information diagnostic of a scene's semantic category, rather than investigating the role that top-down processes play in rapid scene categorization. Thus, most scene gist studies present scenes from multiple categories in randomized sequences.Conversely, in this experiment, we tested whether scene gist recognition is facilitated by sequential expectations.To do this, we created more ecologically valid spatio-temporal "narrative" sequences of images along spatially connected routes from starting points to destinations (e.g., office, hallway, stairwell, sidewalk, parking lot). Scene images were presented one-at-a-time, and based on pilot-testing, briefly flashed (24 ms) and masked (48 ms), followed by selecting the scene category from an 8-AFC array. To reduce predictability of subsequent images, we included subsequences of randomly 1-4 scenes from each category (e.g., 1-4 office images followed by 1-4 hallways, etc.). Critically, scenes were shown in either coherent or randomized narrative sequences to test two competing alternative hypotheses: 1) "Narrative coherence": accuracy is higher for images in coherent narrative sequences because scene category expectations prime representations for to-be-presented scenes; 2) "Feed-forward": accuracy does not differ between coherent and randomized image sequences because it is a purely feed-forward process. Results: images presented in randomized sequences were categorized just as accurately as images presented in coherent sequences, consistent with the "Feed-forward" hypothesis. Furthermore, accuracy did not increase as a function of the number of sequential exposures to a single scene category (e.g., 1-4 office images in a row), suggesting that the mechanisms responsible for perceiving a scene's meaning are so rapid that prior exposures do not support extracting the gist of subsequent scenes. Meeting abstract presented at VSS 2017

Association of rapid scene categorization with a nicotinic acetylcholine receptor gene polymorphism(Summary of Awarded Presentation at the 31st Annual Meeting)

Association of rapid scene categorization with a nicotinic acetylcholine receptor gene polymorphism(Summary of Awarded Presentation at the 31st Annual Meeting)

Two scenes or not two scenes: The effects of stimulus repetition and view-similarity on scene categorization from brief displays.

Previous research suggests that understanding the gist of a scene relies on global structural cues that enable rapid scene categorization. This study used a repetition blindness (RB) paradigm to interrogate the nature of the scene representations used in such rapid categorization. When stimuli are repeated in a rapid serial visual presentation (RSVP) sequence (~10 items/sec), the second occurrence of the repeated item frequently goes unnoticed, a phenomenon that is attributed to a failure to consolidate two conscious episodes (tokens) for a repeatedly activated type. We tested whether RB occurs for different exemplars of the same scene category, which share conceptual and broad structural properties, as well as for identical and mirror-reflected repetitions of the same scene, which additionally share the same local visual details. Across 2 experiments, identical and mirror-image scenes consistently produced a repetition facilitation, rather than RB. There was no convincing evidence of either RB or repetition facilitation for different members of a scene category. These findings indicate that in the first 100-150ms of processing scenes are represented in terms of local visual features, rather than more abstract category-general features, and that, unlike other kinds of stimuli (words or objects), scenes are not susceptible to token individuation failure.

Visual information representation and rapid-scene categorization are simultaneous across cortex: An MEG study

Perceiving the visual world around us requires the brain to represent the features of stimuli and to categorize the stimulus based on these features. Incorrect categorization can result either from errors in visual representation or from errors in processes that lead to categorical choice. To understand the temporal relationship between the neural signatures of such systematic errors, we recorded whole-scalp magnetoencephalography (MEG) data from human subjects performing a rapid-scene categorization task. We built scene category decoders based on (1) spatiotemporally resolved neural activity, (2) spatial envelope (SpEn) image features, and (3) behavioral responses. Using confusion matrices, we tracked how well the pattern of errors from neural decoders could be explained by SpEn decoders and behavioral errors, over time and across cortical areas. Across the visual cortex and the medial temporal lobe, we found that both SpEn and behavioral errors explained unique variance in the errors of neural decoders. Critically, these effects were nearly simultaneous, and most prominent between 100 and 250ms after stimulus onset. Thus, during rapid-scene categorization, neural processes that ultimately result in behavioral categorization are simultaneous and co-localized with neural processes underlying visual information representation.

Processing context: Asymmetric interference of visual form and texture in object and scene interactions

Substantive evidence has demonstrated that scene-centered global image features influence the processing of objects embedded in complex visual scenes. Conversely, a growing body of work suggests that relevant object information may inherently influence diagnostic global scene statistics used in rapid scene categorization. Here, we investigate the potential effects of interference in object–scene perception when attending to form and texture in both simple figure-ground representations and more complex object–background scenes. Results reveal asymmetric interference in the perception of form and texture in object and scene processing: Inconsistent scene texture interfered with the classification of object texture, and inconsistent object form interfered with the classification of scene form, but not vice versa. These findings contribute to our understanding of the interactions between an object and its environment, and further inform our knowledge of the visual features which influence interactivity in object and scene perception.

The influence of segmentation on rapid scene categorization

The influence of segmentation on rapid scene categorization

Distinguishing the roles of color and other surface properties in rapid natural scene categorization: Evidence from ERPs

Distinguishing the roles of color and other surface properties in rapid natural scene categorization: Evidence from ERPs

Does segmentation influence rapid scene categorization?

Does segmentation influence rapid scene categorization?

Does segmentation influence rapid scene categorization

Does segmentation influence rapid scene categorization

Rapid scene categorization: Role of spatial frequency order, accumulation mode and luminance contrast

Visual analysis follows a default, predominantly coarse-to-fine processing sequence. Low spatial frequencies (LSF) are processed more rapidly than high spatial frequencies (HSF), allowing an initial coarse parsing of visual input, prior to analysis of finer information. Our study investigated the influence of spatial frequency processing order, accumulation mode (i.e. how spatial frequency information is received as an input by the visual system, throughout processing), and differences in luminance contrast between spatial frequencies on rapid scene categorization. In Experiment 1, we used sequences composed of six filtered scenes, assembled from LSF to HSF (coarse-to-fine) or from HSF to LSF (fine-to-coarse) to test the effects of spatial frequency order. Spatial frequencies were either successive or additive within sequences to test the effects of spatial frequency accumulation mode. Results showed that participants categorized coarse-to-fine sequences more rapidly than fine-to-coarse sequences, irrespective of spatial frequency accumulation in the sequences. In Experiment 2, we investigated the extent to which differences in luminance contrast rather than in spatial frequency account for the advantage of coarse-to-fine over fine-to-coarse processing. Results showed that both spatial frequencies and luminance contrast account for a predominant coarse-to-fine processing, but that the coarse-to-fine advantage stems mainly from differences in spatial frequencies. Our study cautions against the use of contrast normalization in studies investigating spatial frequency processing. We argue that this type of experimental manipulation can impair the intrinsic properties of a visual stimulus. As the visual system relies on these to enable recognition, bias may be induced in strategies of visual analysis.

An Integrated Saliency Model with Guidance of Eye Movement in Natural Scene Classification

Nature Scene classification is a fundamental problem in image understanding. Human can recognize the scene instantly after only a glance. This is mainly because that our visual attention is easily attracted by the salient objects in scene. And these objects are always representative in the natural scene. It is unclear how humans achieve rapid scene categorization. But this kind of high-level cognitive behavior can be reflected by the eye movement. To identify this ability, we propose a model with the guidance of eye movement. It combines the bag of words (BOW) and spatial pyramid matching (SPM) methods to train and test our model on support vector machine (SVM). The eye movement experiments were employed to validate our model. We found that the subjects could recognize the scenes correctly even if given only a few saliency patches with less than one second. These results suggest that the eye tracking saliency patches play an important role for human scene categorization.

The importance of visual features in rapid scene categorization: evidence from repetition blindness

The importance of visual features in rapid scene categorization: evidence from repetition blindness

The importance of visual features in rapid scene categorization: Evidence from repetition blindness.

The importance of visual features in rapid scene categorization: Evidence from repetition blindness.

A simple rapid categorization model accounts for variations in behavioral responses across rapid scene categorization tasks

A simple rapid categorization model accounts for variations in behavioral responses across rapid scene categorization tasks