A Signature of Orientation Invariance in Human fMRI & (Some) Deep Neural Networks

Abstract

Deep neural network (DNN) models capture the representational geometry of neural responses to different objects (Yamins & Dicarlo, 2016), but do they show the same hallmark invariances as human vision? In the human visual system, orientation invariant representations (consistent responses to an item viewed at different angles) emerge abruptly between V3 and LOC (Morgan & Alvarez, 2014). Here, we ask where (if anywhere) orientation invariance emerges in DNNs, and whether that invariance mirrors the invariance we observe in humans. Using a large battery of DNNs (both trained and randomly initialized, from model zoos including Torchvision and the Taskonomy project), we extract the responses of each layer of each network to a series of 8 stimuli at 5 different degrees of rotation (0-45-90-135-180) – the same stimulus set used in the human fMRI. We then assess the representational similarity of each stimulus to the same stimulus at different rotations (“within-category” similarity) and to different stimuli at the same rotation (“across-category” similarity) in each layer of the network and each area of the brain. We compare the DNN responses directly to the fMRI responses with a second Pearson distance metric across the aggregated similarities (within- and across-category). In the end, this process produces a similarity score for each DNN layer to each area of the brain. Smoothing across the layer-wise similarity scores in each network with a generalized additive model (GAM), we show that in the majority of trained models we test, earlier and intermediate layers of the network are more similar to brain areas with little to no invariance (EVC / OPC) and later layers (usually following a fully connected layer) are more similar to brain areas with strong invariance (LOC / OTC). Overall, our results provide a preliminary signature of human brain-like orientation invariance in deep neural networks.