Scene-selective Cortex Research Articles

Unveiling task-dependent action affordance representations: Insights from scene-selective cortex and deep neural networks

Unveiling task-dependent action affordance representations: Insights from scene-selective cortex and deep neural networks

Combined representation of visual features in the scene-selective cortex

Visual features of separable dimensions conjoin to represent an integrated entity. We investigated how visual features bind to form a complex visual scene. Specifically, we focused on features important for visually guided navigation: direction and distance. Previously, separate works have shown that directions and distances of navigable paths are coded in the occipital place area (OPA). Using functional magnetic resonance imaging (fMRI), we tested how separate features are concurrently represented in the OPA. Participants saw eight types of scenes, in which four of them had one path and the other four had two paths. In single-path scenes, path direction was either to the left or to the right. In double-path scenes, both directions were present. A glass wall was placed in some paths to restrict navigational distance. To test how the OPA represents path directions and distances, we took three approaches. First, the independent-features approach examined whether the OPA codes each direction and distance. Second, the integrated-features approach explored how directions and distances are integrated into path units, as compared to pooled features, using double-path scenes. Finally, the integrated-paths approach asked how separate paths are combined into a scene. Using multi-voxel pattern similarity analysis, we found that the OPA’s representations of single-path scenes were similar to other single-path scenes of either the same direction or the same distance. Representations of double-path scenes were similar to the combination of two constituent single-paths, as a combined unit of direction and distance rather than as a pooled representation of all features. These results show that the OPA combines the two features to form path units, which are then used to build multiple-path scenes. Altogether, these results suggest that visually guided navigation may be supported by the OPA that automatically and efficiently combines multiple features relevant for navigation and represent a navigation file.

Alpha-frequency feedback to early visual cortex orchestrates coherent naturalistic vision.

During naturalistic vision, the brain generates coherent percepts by integrating sensory inputs scattered across the visual field. Here, we asked whether this integration process is mediated by rhythmic cortical feedback. In electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) experiments, we experimentally manipulated integrative processing by changing the spatiotemporal coherence of naturalistic videos presented across visual hemifields. Our EEG data revealed that information about incoherent videos is coded in feedforward-related gamma activity while information about coherent videos is coded in feedback-related alpha activity, indicating that integration is indeed mediated by rhythmic activity. Our fMRI data identified scene-selective cortex and human middle temporal complex (hMT) as likely sources of this feedback. Analytically combining our EEG and fMRI data further revealed that feedback-related representations in the alpha band shape the earliest stages of visual processing in cortex. Together, our findings indicate that the construction of coherent visual experiences relies on cortical feedback rhythms that fully traverse the visual hierarchy.

Dissociable mechanisms for integrating views into places in scene-selective cortex

Dissociable mechanisms for integrating views into places in scene-selective cortex

Combined representation of mid-level visual features in the scene-selective cortex

Combined representation of mid-level visual features in the scene-selective cortex

Spikiness and animacy as potential organizing principles of human ventral visual cortex.

Considerable research has been devoted to understanding the fundamental organizing principles of the ventral visual pathway. A recent study revealed a series of 3-4 topographical maps arranged along the macaque inferotemporal (IT) cortex. The maps articulated a two-dimensional space based on the spikiness and animacy of visual objects, with "inanimate-spiky" and "inanimate-stubby" regions of the maps constituting two previously unidentified cortical networks. The goal of our study was to determine whether a similar functional organization might exist in human IT. To address this question, we presented the same object stimuli and images from "classic" object categories (bodies, faces, houses) to humans while recording fMRI activity at 7 Tesla. Contrasts designed to reveal the spikiness-animacy object space evoked extensive significant activation across human IT. However, unlike the macaque, we did not observe a clear sequence of complete maps, and selectivity for the spikiness-animacy space was deeply and mutually entangled with category-selectivity. Instead, we observed multiple new stimulus preferences in category-selective regions, including functional sub-structure related to object spikiness in scene-selective cortex. Taken together, these findings highlight spikiness as a promising organizing principle of human IT and provide new insights into the role of category-selective regions in visual object processing.

Global and spatially specific representations of 3D scene surface configurations in scene-selective cortex

Global and spatially specific representations of 3D scene surface configurations in scene-selective cortex

Deep neural network models of visual cortex reveal curvature and real-world size as organizing principles of mid-level representation

Mid-level visual features directly support an array of behaviors; thus, they may be critical for understanding the functional organization of visual cortex. However, attempts at characterizing mid-level features have been hampered by the difficulty of describing these features in words—they exist in an “ineffable valley” between the describable patterns of low-level vision (e.g., edges) and the commonsense concepts of visual cognition (e.g., objects). Here we developed a novel approach to identify interpretable emergent properties of mid-level representations in deep neural network (DNN) models of visual cortex. Using this approach, we examined DNN models that were fit to scene-evoked fMRI responses in category-selective regions of visual cortex—specifically, scene-selective cortex (sceneDNN) and object-selective cortex (objectDNN). Our method uses a semantically-guided image-occlusion procedure to systematically characterize how DNN activations are driven by the classes of objects within a scene. We examined the relationship between mid-level features and several object properties that have previously been associated with response preferences in visual cortex: curvature, real-world size, animacy, naturalness, and spatial stability. We found that while mid-level features appear complex and difficult to describe at a surface level, large-scale computational analyses can reveal a latent underlying relationship to interpretable object properties. Specifically, we found that the mid-level representations of the sceneDNN support a latent preference for objects that are boxy and large in real-world size. In contrast, mid-level representations of the objectDNN support a complementary preference for objects that are curvy and small in real-world size. These effects were robust to variations of model hyperparameters and were reproducible across different DNN models. Our findings show that curvature and real-world size are emergent organizing principles of mid-level visual representation, and they suggest that differences in mid-level feature tuning may be critical for understanding the organization of visual cortex into category-selective patches.

Deep-net-derived surface estimations from natural scenes predict voxel responses in scene-selective cortex

Deep-net-derived surface estimations from natural scenes predict voxel responses in scene-selective cortex

Causal neural mechanisms of context-based object recognition.

Objects can be recognized based on their intrinsic features, including shape, color, and texture. In daily life, however, such features are often not clearly visible, for example when objects appear in the periphery, in clutter, or at a distance. Interestingly, object recognition can still be highly accurate under these conditions when objects are seen within their typical scene context. What are the neural mechanisms of context-based object recognition? According to parallel processing accounts, context-based object recognition is supported by the parallel processing of object and scene information in separate pathways. Output of these pathways is then combined in downstream regions, leading to contextual benefits in object recognition. Alternatively, according to feedback accounts, context-based object recognition is supported by (direct or indirect) feedback from scene-selective to object-selective regions. Here, in three pre-registered transcranial magnetic stimulation (TMS) experiments, we tested a key prediction of the feedback hypothesis: that scene-selective cortex causally and selectively supports context-based object recognition before object-selective cortex does. Early visual cortex (EVC), object-selective lateral occipital cortex (LOC), and scene-selective occipital place area (OPA) were stimulated at three time points relative to stimulus onset while participants categorized degraded objects in scenes and intact objects in isolation, in different trials. Results confirmed our predictions: relative to isolated object recognition, context-based object recognition was selectively and causally supported by OPA at 160-200 ms after onset, followed by LOC at 260-300 ms after onset. These results indicate that context-based expectations facilitate object recognition by disambiguating object representations in the visual cortex.

Tracking the Emergence of Location-based Spatial Representations in Human Scene-Selective Cortex.

Scene-selective regions of the human brain form allocentric representations of locations in our environment. These representations are independent of heading direction and allow us to know where we are regardless of our direction of travel. However, we know little about how these location-based representations are formed. Using fMRI representational similarity analysis and linear mixed models, we tracked the emergence of location-based representations in scene-selective brain regions. We estimated patterns of activity for two distinct scenes, taken before and after participants learnt they were from the same location. During a learning phase, we presented participants with two types of panoramic videos: (1) an overlap video condition displaying two distinct scenes (0° and 180°) from the same location and (2) a no-overlap video displaying two distinct scenes from different locations (which served as a control condition). In the parahippocampal cortex (PHC) and retrosplenial cortex (RSC), representations of scenes from the same location became more similar to each other only after they had been shown in the overlap condition, suggesting the emergence of viewpoint-independent location-based representations. Whereas these representations emerged in the PHC regardless of task performance, RSC representations only emerged for locations where participants could behaviorally identify the two scenes as belonging to the same location. The results suggest that we can track the emergence of location-based representations in the PHC and RSC in a single fMRI experiment. Further, they support computational models that propose the RSC plays a key role in transforming viewpoint-independent representations into behaviorally relevant representations of specific viewpoints.

Causal Evidence for a Double Dissociation between Object- and Scene-Selective Regions of Visual Cortex: A Preregistered TMS Replication Study.

Natural scenes are characterized by individual objects as well as by global scene properties such as spatial layout. Functional neuroimaging research has shown that this distinction between object and scene processing is one of the main organizing principles of human high-level visual cortex. For example, object-selective regions, including the lateral occipital complex (LOC), were shown to represent object content (but not scene layout), while scene-selective regions, including the occipital place area (OPA), were shown to represent scene layout (but not object content). Causal evidence for a double dissociation between LOC and OPA in representing objects and scenes is currently limited, however. One TMS experiment, conducted in a relatively small sample (N = 13), reported an interaction between LOC and OPA stimulation and object and scene recognition performance (Dilks et al., 2013). Here, we present a high-powered preregistered replication of this study (N = 72, including male and female human participants), using group-average fMRI coordinates to target LOC and OPA. Results revealed unambiguous evidence for a double dissociation between LOC and OPA: relative to vertex stimulation, TMS over LOC selectively impaired the recognition of objects, while TMS over OPA selectively impaired the recognition of scenes. Furthermore, we found that these effects were stable over time and consistent across individual objects and scenes. These results show that LOC and OPA can be reliably and selectively targeted with TMS, even when defined based on group-average fMRI coordinates. More generally, they support the distinction between object and scene processing as an organizing principle of human high-level visual cortex.SIGNIFICANCE STATEMENT Our daily-life environments are characterized both by individual objects and by global scene properties. The distinction between object and scene processing features prominently in visual cognitive neuroscience, with fMRI studies showing that this distinction is one of the main organizing principles of human high-level visual cortex. However, causal evidence for the selective involvement of object- and scene-selective regions in processing their preferred category is less conclusive. Here, testing a large sample (N = 72) using an established paradigm and a preregistered protocol, we found that TMS over object-selective cortex (lateral occipital complex) selectively impaired object recognition, while TMS over scene-selective cortex (occipital place area) selectively impaired scene recognition. These results provide strong causal evidence for the distinction between object and scene processing in human visual cortex.

Coding of Navigational Distance and Functional Constraint of Boundaries in the Human Scene-Selective Cortex.

For visually guided navigation, the use of environmental cues is essential. Particularly, detecting local boundaries that impose limits to locomotion and estimating their location is crucial. In a series of three fMRI experiments, we investigated whether there is a neural coding of navigational distance in the human visual cortex (both female and male). We used virtual reality software to systematically manipulate the distance from a viewer perspective to different types of a boundary. Using a multivoxel pattern classification employing a linear support vector machine, we found that the occipital place area (OPA) is sensitive to the navigational distance restricted by the transparent glass wall. Further, the OPA was sensitive to a non-crossable boundary only, suggesting an importance of the functional constraint of a boundary. Together, we propose the OPA as a perceptual source of external environmental features relevant for navigation.SIGNIFICANCE STATEMENT One of major goals in cognitive neuroscience has been to understand the nature of visual scene representation in human ventral visual cortex. An aspect of scene perception that has been overlooked despite its ecological importance is the analysis of space for navigation. One of critical computation necessary for navigation is coding of distance to environmental boundaries that impose limit on navigator's movements. This paper reports the first empirical evidence for coding of navigational distance in the human visual cortex and its striking sensitivity to functional constraint of environmental boundaries. Such finding links the paper to previous neurological and behavioral works that emphasized the distance to boundaries as a crucial geometric property for reorientation behavior of children and other animal species.

Boundaries Extend and Contract in Scene Memory Depending on Image Properties.

Boundary extension, a memory distortion in which observers consistently recall a scene with visual information beyond its boundaries, is widely accepted across the psychological sciences as a phenomenon revealing fundamental insight into memory representations [1-3], robust across paradigms [1, 4] and age groups [5-7]. This phenomenon has been taken to suggest that the mental representation of a scene consists of an intermingling of sensory information and a schema that extrapolates the views of a presented scene [8], and it has been used to provide evidence for the role of the neocortex [9] and hippocampus [10, 11] in the schematization of scenes during memory. However, the study of boundary extension has typically focused on object-oriented images that are not representative of our visuospatial world. Here, using a broad set of 1,000 images tested on 2,000 participants in a rapid recognition task, we discover "boundary contraction" as an equally robust phenomenon. Further, image composition largely drives whether extension or contraction is observed-although object-oriented images cause more boundary extension, scene-oriented images cause more boundary contraction. Finally, these effects also occur during drawing tasks, including a task with minimal memory load-when participants copy an image during viewing. Collectively, these results show that boundary extension is not a universal phenomenon and put into question the assumption that scene memory automatically combines visual information with additional context derived from internal schema. Instead, our memory for a scene may be largely driven by its visual composition, with a tendency to extend or contract the boundaries equally likely.

Distinct representations of spatial and categorical relationships across human scene-selective cortex

We represent the locations of places (e.g., the coffee shop on 10th Street vs. the coffee shop on Peachtree Street) so that we can use them as landmarks to orient ourselves while navigating large-scale environments. While several neuroimaging studies have argued that the parahippocampal place area (PPA) represents such navigationally relevant information, evidence from other studies suggests otherwise, leaving this issue unresolved. Here we hypothesize that the PPA is, in fact, not well suited to recognize specific landmarks in the environment (e.g., the coffee shop on 10th Street), but rather is involved in recognizing the general category membership of places (e.g., a coffee shop, regardless of its location). Using fMRI multivoxel pattern analysis, we directly test this hypothesis. If the PPA represents landmark information, then it must be able to discriminate between 2 places of the same category, but in different locations. Instead, if the PPA represents general category information (as hypothesized here), then it will not represent the location of a particular place, but only the category of the place. As predicted, we found that the PPA represents 2 buildings from the same category, but in different locations, as more similar than 2 buildings from different categories, but in the same location. In contrast, another scene-selective region of cortex, the retrosplenial complex (RSC), showed the exact opposite pattern of results. Such a double dissociation suggests distinct neural systems involved in categorizing and navigating our environment, including the PPA and RSC, respectively.

Scene feature preferences found in scene selective cortex

Scene feature preferences found in scene selective cortex

Human V2A: A map of the peripheral visual hemifield with functional connections to scene-selective cortex.

Humans can recognize a scene in the blink of an eye. This gist-based visual scene perception is thought to be underpinned by specialized visual processing emphasizing the visual periphery at a cortical locus relatively low in the visual processing hierarchy. Using wide-field retinotopic mapping and population receptive field (pRF) modeling, we identified a new visual hemifield map anterior of area V2d and inferior to area V6, which we propose to call area V2A. Based on its location relative to other visual areas, V2A may correspond to area 23V described in nonhuman primates. The pRF analysis revealed unique receptive field properties for V2A: a large (FWHM ∼23°) and constant receptive field size across the central ∼70° of the visual field. Resting-state fMRI connectivity analysis further suggests that V2A is ideally suited to quickly feed the scene-processing network with information that is not biased towards the center of the visual field. Our findings not only indicate a likely cortical locus for the initial stages of gist-based visual scene perception, but also suggest a reappraisal of the organization of human dorsomedial occipital cortex with a strip of separate hemifield representations anterior to the early visual areas (V1, V2d, and V3d).

Scene content is predominantly conveyed by high spatial frequencies in scene-selective visual cortex

Scene content is predominantly conveyed by high spatial frequencies in scene-selective visual cortex

Coding of navigational distance in the visual scene-selective cortex

Coding of navigational distance in the visual scene-selective cortex

Representational differences between line drawings and photographs of natural scenes: A dissociation between multi-voxel pattern analysis and repetition suppression

Distributed representations of scene categories are consistent between color photographs (CPs) and line drawings (LDs) in the parahippocampal place area (PPA) and the retrosplenial cortex (RSC), as shown using multi-voxel pattern analysis (MVPA). Here, we used repetition suppression (RS) to further investigate the degree of representational convergence between CPs and LDs of natural scenes. MVPA and RS can capture different aspects of visual representations, and RS may prove useful in elucidating important differences in the representations of CPs and LDs of natural scenes. We performed an event-related fMRI experiment, including image-repetitions either within-type (i.e., CP to CP or LD to LD) or between-types (CP to LD, LD to CP). We found significant RS for within-type repetitions in PPA, RSC and the occipital place area (OPA), but did not observe RS for between-types repetitions. By contrast, scene categories were decodable from activity patterns evoked by both CPs and LDs using SVM classification for both within-type decoding and between-types cross-decoding. We conclude that there are representational differences between CPs and LDs in scene-selective cortex despite a category-level correspondence.