Decoding individual natural scene representations during perception and imagery

Matthew R Johnson,Marcia K Johnson

doi:10.3389/fnhum.2014.00059

Abstract

We used a multi-voxel classification analysis of functional magnetic resonance imaging (fMRI) data to determine to what extent item-specific information about complex natural scenes is represented in several category-selective areas of human extrastriate visual cortex during visual perception and visual mental imagery. Participants in the scanner either viewed or were instructed to visualize previously memorized natural scene exemplars, and the neuroimaging data were subsequently subjected to a multi-voxel pattern analysis (MVPA) using a support vector machine (SVM) classifier. We found that item-specific information was represented in multiple scene-selective areas: the occipital place area (OPA), parahippocampal place area (PPA), retrosplenial cortex (RSC), and a scene-selective portion of the precuneus/intraparietal sulcus region (PCu/IPS). Furthermore, item-specific information from perceived scenes was re-instantiated during mental imagery of the same scenes. These results support findings from previous decoding analyses for other types of visual information and/or brain areas during imagery or working memory, and extend them to the case of visual scenes (and scene-selective cortex). Taken together, such findings support models suggesting that reflective mental processes are subserved by the re-instantiation of perceptual information in high-level visual cortex. We also examined activity in the fusiform face area (FFA) and found that it, too, contained significant item-specific scene information during perception, but not during mental imagery. This suggests that although decodable scene-relevant activity occurs in FFA during perception, FFA activity may not be a necessary (or even relevant) component of one's mental representation of visual scenes.

Highlights

Current models of working memory and related reflective activities suggest that active representations are maintained via control signals originating in heteromodal association areas that re-instantiate neural activity in sensory cortex that was first engaged when an item was initially perceived (Petrides, 1994; Kosslyn et al, 2001; Curtis and D’Esposito, 2003; Ruchkin et al, 2003; Pasternak and Greenlee, 2005; Ranganath and D’Esposito, 2005)
We examined activity in four scene-selective a priori regions of interest (ROIs) (OPA, parahippocampal place area (PPA), retrosplenial cortex (RSC), and precuneus/intraparietal sulcus region (PCu/IPS), as noted in the Materials and Methods section; see Figure 1C), as well as fusiform face area (FFA), and used an support vector machine (SVM) classification algorithm to determine whether each ROI contained information that allowed the classifier to distinguish between each pair of conditions
Classifiers testing for re-instantiation performed above chance for scene-selective cortex as a whole (AUCs: Experiment 1 = 0.553, Experiment 2 = 0.561)

Summary

Introduction

Current models of working memory and related reflective activities (e.g., mental imagery) suggest that active representations are maintained via control signals originating in heteromodal association areas (e.g., prefrontal cortex) that re-instantiate neural activity in sensory cortex that was first engaged when an item was initially perceived (Petrides, 1994; Kosslyn et al, 2001; Curtis and D’Esposito, 2003; Ruchkin et al, 2003; Pasternak and Greenlee, 2005; Ranganath and D’Esposito, 2005) Consistent with these models, earlier neuroimaging studies observed category-related activity in category-selective extrastriate (CSE) visual areas such as fusiform face area (FFA; Kanwisher et al, 1997; McCarthy et al, 1997) and parahippocampal place area (PPA; Epstein and Kanwisher, 1998) when individuals maintained representations of items from the relevant category during visual working memory (Druzgal and D’Esposito, 2003; Postle et al, 2003; Ranganath et al, 2004). Multi-voxel pattern analysis (MVPA) is one method that can assess such patterns

Objectives

Methods

Results

Discussion

Conclusion