Human Ventral Visual Pathway Research Articles

A Convolutional Neural Network Interpretable Framework for Human Ventral Visual Pathway Representation

Recently, convolutional neural networks (CNNs) have become the best quantitative encoding models for capturing neural activity and hierarchical structure in the ventral visual pathway. However, the weak interpretability of these black-box models hinders their ability to reveal visual representational encoding mechanisms. Here, we propose a convolutional neural network interpretable framework (CNN-IF) aimed at providing a transparent interpretable encoding model for the ventral visual pathway. First, we adapt the feature-weighted receptive field framework to train two high-performing ventral visual pathway encoding models using large-scale functional Magnetic Resonance Imaging (fMRI) in both goal-driven and data-driven approaches. We find that network layer-wise predictions align with the functional hierarchy of the ventral visual pathway. Then, we correspond feature units to voxel units in the brain and successfully quantify the alignment between voxel responses and visual concepts. Finally, we conduct Network Dissection along the ventral visual pathway including the fusiform face area (FFA), and discover variations related to the visual concept of `person'. Our results demonstrate the CNN-IF provides a new perspective for understanding encoding mechanisms in the human ventral visual pathway, and the combination of ante-hoc interpretable structure and post-hoc interpretable approaches can achieve fine-grained voxel-wise correspondence between model and brain. The source code is available at: https://github.com/BIT-YangLab/CNN-IF.

Comparing the Dominance of Color and Form Information across the Human Ventral Visual Pathway and Convolutional Neural Networks.

Color and form information can be decoded in every region of the human ventral visual hierarchy, and at every layer of many convolutional neural networks (CNNs) trained to recognize objects, but how does the coding strength of these features vary over processing? Here, we characterize for these features both their absolute coding strength-how strongly each feature is represented independent of the other feature-and their relative coding strength-how strongly each feature is encoded relative to the other, which could constrain how well a feature can be read out by downstream regions across variation in the other feature. To quantify relative coding strength, we define a measure called the form dominance index that compares the relative influence of color and form on the representational geometry at each processing stage. We analyze brain and CNN responses to stimuli varying based on color and either a simple form feature, orientation, or a more complex form feature, curvature. We find that while the brain and CNNs largely differ in how the absolute coding strength of color and form vary over processing, comparing them in terms of their relative emphasis of these features reveals a striking similarity: For both the brain and for CNNs trained for object recognition (but not for untrained CNNs), orientation information is increasingly de-emphasized, and curvature information is increasingly emphasized, relative to color information over processing, with corresponding processing stages showing largely similar values of the form dominance index.

The Privileged Status of Peer Faces: Subordinate-level Neural Representations of Faces in Emerging Adults.

Faces can be represented at a variety of different subordinate levels (e.g., race) that can become "privileged" for visual recognition in perceivers and is reflected as patterns of biases (e.g., own-race bias). The mechanisms encoding privileged status are likely varied, making it difficult to predict how neural systems represent subordinate-level biases in face processing. Here, we investigate the neural basis of subordinate-level representations of human faces in the ventral visual pathway, by leveraging recent behavioral findings indicating the privileged nature of peer faces in identity recognition for adolescents and emerging adults (i.e., ages 18-25 years). We tested 166 emerging adults in a face recognition paradigm and a subset of 31 of these participants in two fMRI task paradigms. We showed that emerging adults exhibit a peer bias in face recognition behavior, which indicates a privileged status for a subordinate-level category of faces that is not predicted based on experience alone. This privileged status of peer faces is supported by multiple neural mechanisms within the ventral visual pathway, including enhanced neural magnitude and neural size in the FFA1, which is a critical part of the face-processing network, that fundamentally supports the representations of subordinate-level categories of faces (in terms of developmental stage). These findings demonstrate organizational principles that the human ventral visual pathway uses to privilege relevant social information in face representations, which is essential for navigating human social interactions. It will be important to understand whether similar mechanisms support representations of other subordinate-level categories like race and gender.

Invariant representation of physical stability in the human brain.

Successful engagement with the world requires the ability to predict what will happen next. Here, we investigate how the brain makes a fundamental prediction about the physical world: whether the situation in front of us is stable, and hence likely to stay the same, or unstable, and hence likely to change in the immediate future. Specifically, we ask if judgments of stability can be supported by the kinds of representations that have proven to be highly effective at visual object recognition in both machines and brains, or instead if the ability to determine the physical stability of natural scenes may require generative algorithms that simulate the physics of the world. To find out, we measured responses in both convolutional neural networks (CNNs) and the brain (using fMRI) to natural images of physically stable versus unstable scenarios. We find no evidence for generalizable representations of physical stability in either standard CNNs trained on visual object and scene classification (ImageNet), or in the human ventral visual pathway, which has long been implicated in the same process. However, in frontoparietal regions previously implicated in intuitive physical reasoning we find both scenario-invariant representations of physical stability, and higher univariate responses to unstable than stable scenes. These results demonstrate abstract representations of physical stability in the dorsal but not ventral pathway, consistent with the hypothesis that the computations underlying stability entail not just pattern classification but forward physical simulation.

Representation of color, form, and their conjunction across the human ventral visual pathway

Despite decades of research, our understanding of the relationship between color and form processing in the primate ventral visual pathway remains incomplete. Using fMRI multivoxel pattern analysis, we examined coding of color and form, using a simple form feature (orientation) and a mid-level form feature (curvature), in human ventral visual processing regions. We found that both color and form could be decoded from activity in early visual areas V1 to V4, as well as in the posterior color-selective region and shape-selective regions in ventral and lateral occipitotemporal cortex defined based on their univariate selectivity to color or shape, respectively (the central color region only showed color but not form decoding). Meanwhile, decoding biases towards one feature or the other existed in the color- and shape-selective regions, consistent with their univariate feature selectivity reported in past studies. Additional extensive analyses show that while all these regions contain independent (linearly additive) coding for both features, several early visual regions also encode the conjunction of color and the simple, but not the complex, form feature in a nonlinear, interactive manner. Taken together, the results show that color and form are encoded in a biased distributed and largely independent manner across ventral visual regions in the human brain.

The relative coding strength of color and form in the human ventral visual pathway and convolutional neural networks

The relative coding strength of color and form in the human ventral visual pathway and convolutional neural networks

Predicting Identity-Preserving Object Transformations across the Human Ventral Visual Stream.

In everyday life, we have no trouble categorizing objects varying in position, size, and orientation. Previous fMRI research shows that higher-level object processing regions in the human lateral occipital cortex may link object responses from different affine states (i.e., size and viewpoint) through a general linear mapping function capable of predicting responses to novel objects. In this study, we extended this approach to examine the mapping for both Euclidean (e.g., position and size) and non-Euclidean (e.g., image statistics and spatial frequency) transformations across the human ventral visual processing hierarchy, including areas V1, V2, V3, V4, ventral occipitotemporal cortex, and lateral occipitotemporal cortex. The predicted pattern generated from a linear mapping function could capture a significant amount of the changes associated with the transformations throughout the ventral visual stream. The derived linear mapping functions were not category independent as performance was better for the categories included than those not included in training and better between two similar versus two dissimilar categories in both lower and higher visual regions. Consistent with object representations being stronger in higher than in lower visual regions, pattern selectivity and object category representational structure were somewhat better preserved in the predicted patterns in higher than in lower visual regions. There were no notable differences between Euclidean and non-Euclidean transformations. These findings demonstrate a near-orthogonal representation of object identity and these nonidentity features throughout the human ventral visual processing pathway with these nonidentity features largely untangled from the identity features early in visual processing.SIGNIFICANCE STATEMENT Presently we still do not fully understand how object identity and nonidentity (e.g., position, size) information are simultaneously represented in the primate ventral visual system to form invariant representations. Previous work suggests that the human lateral occipital cortex may be linking different affine states of object representations through general linear mapping functions. Here, we show that across the entire human ventral processing pathway, we could link object responses in different states of nonidentity transformations through linear mapping functions for both Euclidean and non-Euclidean transformations. These mapping functions are not identity independent, suggesting that object identity and nonidentity features are represented in a near rather than a completely orthogonal manner.

Topography of Visual Features in the Human Ventral Visual Pathway.

Visual object recognition in humans and nonhuman primates is achieved by the ventral visual pathway (ventral occipital-temporal cortex, VOTC), which shows a well-documented object domain structure. An on-going question is what type of information is processed in the higher-order VOTC that underlies such observations, with recent evidence suggesting effects of certain visual features. Combining computational vision models, fMRI experiment using a parametric-modulation approach, and natural image statistics of common objects, we depicted the neural distribution of a comprehensive set of visual features in the VOTC, identifying voxel sensitivities with specific feature sets across geometry/shape, Fourier power, and color. The visual feature combination pattern in the VOTC is significantly explained by their relationships to different types of response-action computation (fight-or-flight, navigation, and manipulation), as derived from behavioral ratings and natural image statistics. These results offer a comprehensive visual feature map in the VOTC and a plausible theoretical explanation as a mapping onto different types of downstream response-action systems.

Examining the Coding Strength of Object Identity and Nonidentity Features in Human Occipito-Temporal Cortex and Convolutional Neural Networks.

A visual object is characterized by multiple visual features, including its identity, position and size. Despite the usefulness of identity and nonidentity features in vision and their joint coding throughout the primate ventral visual processing pathway, they have so far been studied relatively independently. Here in both female and male human participants, the coding of identity and nonidentity features was examined together across the human ventral visual pathway. The nonidentity features tested included two Euclidean features (position and size) and two non-Euclidean features (image statistics and spatial frequency (SF) content of an image). Overall, identity representation increased and nonidentity feature representation decreased along the ventral visual pathway, with identity outweighing the non-Euclidean but not the Euclidean features at higher levels of visual processing. In 14 convolutional neural networks (CNNs) pretrained for object categorization with varying architecture, depth, and with/without recurrent processing, nonidentity feature representation showed an initial large increase from early to mid-stage of processing, followed by a decrease at later stages of processing, different from brain responses. Additionally, from lower to higher levels of visual processing, position became more underrepresented and image statistics and SF became more overrepresented compared with identity in CNNs than in the human brain. Similar results were obtained in a CNN trained with stylized images that emphasized shape representations. Overall, by measuring the coding strength of object identity and nonidentity features together, our approach provides a new tool for characterizing feature coding in the human brain and the correspondence between the brain and CNNs.SIGNIFICANCE STATEMENT This study examined the coding strength of object identity and four types of nonidentity features along the human ventral visual processing pathway and compared brain responses with those of 14 convolutional neural networks (CNNs) pretrained to perform object categorization. Overall, identity representation increased and nonidentity feature representation decreased along the ventral visual pathway, with some notable differences among the different nonidentity features. CNNs differed from the brain in a number of aspects in their representations of identity and nonidentity features over the course of visual processing. Our approach provides a new tool for characterizing feature coding in the human brain and the correspondence between the brain and CNNs.

Neural representations of perceptual color experience in the human ventral visual pathway

Color is a perceptual construct that arises from neural processing in hierarchically organized cortical visual areas. Previous research, however, often failed to distinguish between neural responses driven by stimulus chromaticity versus perceptual color experience. An unsolved question is whether the neural responses at each stage of cortical processing represent a physical stimulus or a color we see. The present study dissociated the perceptual domain of color experience from the physical domain of chromatic stimulation at each stage of cortical processing by using a switch rivalry paradigm that caused the color percept to vary over time without changing the retinal stimulation. Using functional MRI (fMRI) and a model-based encoding approach, we found that neural representations in higher visual areas, such as V4 and VO1, corresponded to the perceived color, whereas responses in early visual areas V1 and V2 were modulated by the chromatic light stimulus rather than color perception. Our findings support a transition in the ascending human ventral visual pathway, from a representation of the chromatic stimulus at the retina in early visual areas to responses that correspond to perceptually experienced colors in higher visual areas.

Examining the representational content of perirhinal cortex and posterior ventral visual pathway regions when maintenance of visual information is interrupted

The perirhinal cortex (PRC) is known to support recognition memory, working memory, and perception for objects. Often, information must be maintained in working memory in the face of ongoing visual perception, raising the question of how PRC and other regions supporting object representation deal with this conflict. Here, we used functional MRI to examine the representational content of human ventral visual pathway (VVP) regions, including perirhinal cortex (PRC), during a visual delayed match-to-sample task. Critically, interfering items from a different stimulus category to the target were presented to participants during the maintenance phase, creating conflict between the contents of working memory and ongoing perception. Using representational similarity analysis, we compared information content across study, interference and test phases to models that differed in the extent to which they predicted that information content would be maintained during the delay period and perturbed by interference. This revealed that lateral occipital and fusiform regions are best fit by models that reflect the stimulus content of the item currently being viewed, whereas PRC reflects the information content of the studied item, even when an interfering stimulus is presented during the delay. This division of labor within the VVP sheds light on the specialized nature of representations in VVP regions, including PRC. Our findings support a representational hierarchical understanding of medial temporal lobe function which posits that representations at the most anterior aspect of the VVP are more robust to ongoing perceptual activity.

A step toward understanding the human ventral visual pathway.

The human ventral visual pathway is implicated in higher order form processing, but the organizational principles within this region are not yet well understood. Recently, Lafer-Sousa, Conway, and Kanwisher (J Neurosci 36: 1682-1697, 2016) used functional magnetic resonance imaging to demonstrate that functional responses in the human ventral visual pathway share a broad homology with the those in macaque inferior temporal cortex, providing new evidence supporting the validity of the macaque as a model of the human visual system in this region. In addition, these results give new clues for understanding the organizational principles within the ventral visual pathway and the processing of higher order color and form, suggesting new avenues for research into this cortical region.

A gradual increase of face-selectivity along the human ventral visual pathway: evidence from intracerebral recordings with fast periodic visual stimulation

Face perception involves a large set of regions distributed along the ventral temporal cortex and thus represents an interesting model to help understanding how visual information is processed along the ventral visual pathway. Here we shed light on the functional organization of this cortical network by recording focal intracerebral electroencephalogram in 20 epileptic patients implanted with depth electrodes. We objectively defined and quantified face-selective responses in the frequency domain by means of fast periodic visual stimulation with natural images (Rossion et al., in press, JOV). Patients were presented with various object categories (houses, animals, plants, etc.) at a base frequency of 6 Hz (6 images/second) with variable face stimuli interleaved in this sequence at regular intervals of 5 stimuli (face frequency = 6 Hz/5 = 1.2 Hz) (Figure 1A). They were instructed to detect colour changes of the fixation cross. Robust face-selective responses occurring exactly at the 1.2 Hz oddball frequency and its harmonics (2.4, 3.6, etc.) were found in the lateral occipital cortex and in the fusiform gyrus (FG), but also in the antero-inferior temporal cortex (AIT: anterior collateral and occipito-temporal sulci) and in the temporal pole (TP). Interestingly, there was a posterior-anterior gradient of face-specificity: face/base frequency ratio increased along the ventral temporal regions (see Figure 1B for typical responses in 3 patients). Although the magnitude of the face-selective responses were the largest in regions that also responded to all visual stimuli (FG), pure response to faces (oddball responses in the absence of any 6 Hz general visual responses) were observed only in the AIT and in the TP (e.g., patients 2 and 3, Figure 1B). These findings point to a sharpening of face-selectivity along the ventral visual pathway and reveal that exclusive response to faces can be found at a macroscopic (cell populations) level of cortical organization. Meeting abstract presented at VSS 2015.

Representation of Contextually Related Multiple Objects in the Human Ventral Visual Pathway

Real-world scenes usually contain a set of cluttered and yet contextually related objects. Here we used fMRI to investigate where and how contextually related multiple objects were represented in the human ventral visual pathway. Specifically, we measured the responses in face-selective and body-selective regions along the ventral pathway when faces and bodies were presented either simultaneously or in isolation. We found that, in the posterior regions, the response for the face and body pair was the weighted average response for faces and bodies presented in isolation. In contrast, the anterior regions encoded the face and body pair in a mutually facilitative fashion, with the response for the pair significantly higher than that for its constituent objects. Furthermore, in the right fusiform face area, the face and body pair was represented as one inseparable object, possibly to reduce perceptual load and increase representation efficiency. Therefore, our study suggests that the visual system uses a hierarchical representation scheme to process multiple objects in natural scenes: the average mechanism in posterior regions helps retaining information of individual objects in clutter, whereas the nonaverage mechanism in the anterior regions uses the contextual information to optimize the representation for multiple objects.

Data-driven functional clustering reveals dominance of face, place, and body selectivity in the ventral visual pathway

Regions selective for faces, places, and bodies feature prominently in the literature on the human ventral visual pathway. Are selectivities for these categories in fact the most robust response profiles in this pathway, or is their prominence an artifact of biased sampling of the hypothesis space in prior work? Here we use a data-driven structure discovery method that avoids the assumptions built into most prior work by 1) giving equal consideration to all possible response profiles over the conditions tested, 2) relaxing implicit anatomical constraints (that important functional profiles should manifest themselves in spatially contiguous voxels arising in similar locations across subjects), and 3) testing for dominant response profiles over images, rather than categories, thus enabling us to discover, rather than presume, the categories respected by the brain. Even with these assumptions relaxed, face, place, and body selectivity emerge as dominant in the ventral stream.

Fast and Automatic Activation of an Abstract Representation of Money in the Human Ventral Visual Pathway

Money, when used as an incentive, activates the same neural circuits as rewards associated with physiological needs. However, unlike physiological rewards, monetary stimuli are cultural artifacts: how are monetary stimuli identified in the first place? How and when does the brain identify a valid coin, i.e. a disc of metal that is, by social agreement, endowed with monetary properties? We took advantage of the changes in the Euro area in 2002 to compare neural responses to valid coins (Euros, Australian Dollars) with neural responses to invalid coins that have lost all monetary properties (French Francs, Finnish Marks). We show in magneto-encephalographic recordings, that the ventral visual pathway automatically distinguishes between valid and invalid coins, within only ∼150 ms. This automatic categorization operates as well on coins subjects were familiar with as on unfamiliar coins. No difference between neural responses to scrambled controls could be detected. These results could suggest the existence of a generic, all-purpose neural representation of money that is independent of experience. This finding is reminiscent of a central assumption in economics, money fungibility, or the fact that a unit of money is substitutable to another. From a neural point of view, our findings may indicate that the ventral visual pathway, a system previously thought to analyze visual features such as shape or color and to be influenced by daily experience, could also able to use conceptual attributes such as monetary validity to categorize familiar as well as unfamiliar visual objects. The symbolic abilities of the posterior fusiform region suggested here could constitute an efficient neural substrate to deal with culturally defined symbols, independently of experience, which probably fostered money's cultural emergence and success.

Transformation from image-based to perceptual representation of materials along the human ventral visual pathway

Every object in the world has its own surface quality that is a reflection of the material from which the object is made. We can easily identify and categorize materials (wood, metal, fabric etc.) at a glance, and this ability enables us to decide how to interact appropriately with these objects. Little is known, however, about how materials are represented in the brain, or how that representation is related to material perception or the physical properties of material surface. By combining multivoxel pattern analysis of functional magnetic resonance imaging data with perceptual and image-based physical measures of material properties, we found that the way visual information about materials is coded gradually changes from an image-based representation in early visual areas to a perceptual representation in the ventral higher-order visual areas. We suggest that meaningful information about multimodal aspects of real-world materials reside in the ventral cortex around the fusiform gyrus, where it can be utilized for categorization of materials.

Linearity of the fMRI response in category-selective regions of human visual cortex.

The goal of this study was to determine the linearity of the blood oxygen level-dependent (BOLD) response, as measured by functional magnetic resonance imaging (fMRI), in category-selective regions of human visual cortex. We defined regions of the temporal lobe that were selective to faces (fusiform face area, FFA) and places (parahippocampal place area, PPA). We then determined the linearity of the BOLD response in these regions to their preferred and nonpreferred stimuli. First, we tested the principle of scaling. As we increased the visibility of the stimulus, there was a corresponding linear increase in the fMRI signal in the FFA and PPA to their preferred stimulus (face and place, respectively). In contrast, responses in the FFA and PPA to the nonpreferred stimulus did not conform to the principle of scaling. Next, we asked whether the fMRI response in these regions of visual cortex conformed to the principle of additivity. To assess this, we determined whether the response to a long stimulus block could be predicted by adding the response to multiple shorter duration blocks. Although the fMRI response in the FFA and PPA was generally linear to the preferred stimulus, a more nonlinear response was apparent to the nonpreferred stimulus. In conclusion, the linearity of the BOLD response in the human ventral visual pathway varied across cortical region and stimulus category. This suggests that measures of linearity may provide a useful indication of neural selectivity in the brain.

Uncertainty and Invariance in the Human Visual Cortex

The way in which input noise perturbs the behavior of a system depends on the internal processing structure of the system. In visual psychophysics, there is a long tradition of using external noise methods (i.e., adding noise to visual stimuli) as tools for system identification. Here, we demonstrate that external noise affects processing of visual scenes at different cortical areas along the human ventral visual pathway, from retinotopic regions to higher occipitotemporal areas implicated in visual shape processing. We found that when the contrast of the stimulus was held constant, the further away from the retinal input a cortical area was the more its activity, as measured with functional magnetic resonance imaging (fMRI), depended on the signal-to-noise ratio (SNR) of the visual stimulus. A similar pattern of results was observed when trials with correct and incorrect responses were analyzed separately. We interpret these findings by extending signal detection theory to fMRI data analysis. This approach reveals the sequential ordering of decision stages in the cortex by exploiting the relation between fMRI response and stimulus SNR. In particular, our findings provide novel evidence that occipitotemporal areas in the ventral visual pathway form a cascade of decision stages with increasing degree of signal uncertainty and feature invariance.

Object-form topology in the ventral temporal lobe: Response to

In her thoughtful comment on the functional organization of ventral temporal cortex, Isabel Gauthier clearly outlines the hypotheses that have been proposed and are currently being investigated with functional brain imaging [Gauthier, I. (2000) What constrains the organization of the ventral temporal cortex? Trends Cognit. Sci. 4, 1–2] 1 Gauthier I. What constrains the organization of the ventral temporal cortex?. Trends Cognit. Sci. 2000; 4: 1-2 Abstract Full Text Full Text PDF PubMed Scopus (83) Google Scholar . One of those models, which we call the object form topology hypothesis, was proposed by us in a recent paper 2 Ishai A. et al. Distributed representation of objects in the human ventral visual pathway. Proc. Natl. Acad. Sci. U. S. A. 1999; 96: 9379-9384 Crossref PubMed Scopus (705) Google Scholar that motivated Gauthier’s comment. In this reply, we would like to explain why we have proposed the object-form topology hypothesis and clarify what we believe are its most salient aspects.