High-performing computational models of visual cortex are marked by high intrinsic dimensionality

Abstract

A central goal of cognitive neuroscience is to understand the computational properties of neural representations in visual cortex. Above and beyond the information content or visual features encoded in neural populations, we wish to understand the encoding format itself and the computations it subserves. These questions are at the heart of research into computational properties such as sparsity, dimensionality, manifold geometry, invariance, and dynamics in both biological and artificial neural networks. Here we investigate the computational properties of representations across human visual cortex using fMRI encoding models. We first fit voxelwise encoding models to predict fMRI responses to images of natural objects and scenes using both hand-engineered feature detectors (e.g. edge detectors) and pre-trained and untrained convolutional neural networks (CNNs). We then performed statistical analyses of the encoding model regressors to explore how performance is affected by several computational properties, including mixed selectivity and intrinsic dimensionality. We searched for the properties that correlated best with encoding accuracy. We find evidence for high intrinsic dimensionality as an important driver of encoding model performance across the ventral and dorsal visual streams. Furthermore, we find that several mechanisms for increasing intrinsic dimensionality lead to similar improvements in encoding model performance, including division normalization, multiplicative interaction, random nonlinear projection, and supervised learning for image classification. Thus, it appears that intrinsic dimensionality itself best explains the improvements of these varied representational mechanisms. Together, our work explores a methodology for investigating the computational properties of neural representations through an ensemble of encoding models and statistical analyses of their computational properties. We use this approach to identify high intrinsic dimensionality as a key predictor of encoding model performance, suggesting that biological vision relies on representations that are highly efficient, and may allow for the linear separability of a large number of visual properties across different tasks.