Organizational motifs of cortical responses to objects emerge in topographic projections of deep neural networks

Abstract

Visual system responses to object images have a systematic topography along the two-dimensional cortex, with selective regions for faces and places located meaningfully within a larger weaker organization of animacy and object size. Interestingly, standard deep convolutional neural networks do not have explicit topography or category-specialized mechanisms, but nonetheless learn feature tuning that has a significant correspondence with neural responses. Here we developed a method to explore the implicit topography within these deep neural network representations, by smoothly mapping the learned feature spaces with a simulated two-dimensional cortex trained using self-organizing principles. We projected a 20x20 grid of map units into the 4096-dimensional feature space of the FC6 layer of object-trained AlexNet, where nearby units on the map have similar tuning. This simulated cortex reflects a smooth spatialized representation of the data manifold in this layer, while still capturing much of the representational geometry of the original deep net layer (r=0.67). Next, we calculated the simulated cortical responses to several localizer image sets. We found a large-scale organization of animate vs inanimate response preferences. Further, clusters of face-selective and place-selective units were evident, even though the object-trained Alexnet wasn’t optimized with specialized mechanisms for these categories. As in the human brain, these emergent face regions were within animate preferring zones, while place regions were generally within inanimate zones. Finally, these results were not obtained in randomly initialized projections. Overall, these topographic projections reveal that some of the known large-scale organizational motifs of tuning properties across the human occipitotemporal cortex are implicit in the representational structure learned by deep convolutional neural networks. Broadly, this work provides evidence that the entire object-selective cortex may reflect a smoothly mapped, integrated feature space (Prince & Konkle, 2020), and introduces a new method to link hypothesized representational spaces and spatialized cortical responses.