Scene themes, natural scene structures, and spatial statistics for scene vision

Abstract

Humans can grasp the gist of complex natural scenes quickly and remember rich detail in thousands of scenes viewed for only a brief period. However, little is known about the underlying neural processing. Both global and local features have been proposed to account for this rapid scene perception. Global features such as the gist descriptor encode structures of whole scenes but leave out local visual features and their spatial relationships. Local features such as scale-invariant feature transform encode statistics of local structures but leave out scene attributes at intermediate and large scales. In this study, we propose a theoretical framework of neural codes of natural scenes and scene perception. In this framework, 1) a visual scene is a sample from a probability distribution (PD) of natural scenes in terms of scene themes that specify object categories and their spatial layouts; 2) each scene theme is probabilistically represented by a set of natural scene structures (NSSs), which are patterns of co-occurrences of basic features, and their spatial arrangements; 3) neuronal codes of natural scenes are probabilities evaluated on the basis of scene themes and NSSs and their spatial arrangements; 4) scene perception is generated via statistical inference that involves extensive interaction between bottom-up and top-down processing based on PDs of natural scenes; and 5) scene themes and NSSs and their spatial arrangements facilitate other visual tasks, including space and scene memory, visual search, and object detection. To test this framework, we compiled scene themes and NSSs in several large datasets of scene categories and developed the proposed PDs of natural scenes. We then used these PDs to categorize natural scenes and found that the categorization accuracy is comparable to or better than the state-of–the-art models. This result calls for studies to test the psychophysical and neurobiological implications of this framework. Meeting abstract presented at VSS 2015