Visual prototypes in the ventral stream are attuned to complexity and gaze behavior

Olivia Rose,James Johnson,Binxu Wang,Carlos R Ponce

doi:10.1038/s41467-021-27027-8

Olivia Rose, James Johnson + Show 2 more

Open Access

https://doi.org/10.1038/s41467-021-27027-8

Copy DOI

Abstract

Early theories of efficient coding suggested the visual system could compress the world by learning to represent features where information was concentrated, such as contours. This view was validated by the discovery that neurons in posterior visual cortex respond to edges and curvature. Still, it remains unclear what other information-rich features are encoded by neurons in more anterior cortical regions (e.g., inferotemporal cortex). Here, we use a generative deep neural network to synthesize images guided by neuronal responses from across the visuocortical hierarchy, using floating microelectrode arrays in areas V1, V4 and inferotemporal cortex of two macaque monkeys. We hypothesize these images (“prototypes”) represent such predicted information-rich features. Prototypes vary across areas, show moderate complexity, and resemble salient visual attributes and semantic content of natural images, as indicated by the animals’ gaze behavior. This suggests the code for object recognition represents compressed features of behavioral relevance, an underexplored aspect of efficient coding.

Highlights

Theories of efficient coding suggested the visual system could compress the world by learning to represent features where information was concentrated, such as contours
It was found that neurons in early visual cortex encode information about contours and corners[4], with neurons representing Gabor-like filters of different orientations and spatial frequencies[5]
We recorded from 128 unique cortical sites in two male monkeys (Macaca mulatta) in regions conventionally described as the border of V1/V213, V414,15, and posterior/central IT16 (Fig. 1a, Methods)

Summary

Introduction

Theories of efficient coding suggested the visual system could compress the world by learning to represent features where information was concentrated, such as contours. Prototypes vary across areas, show moderate complexity, and resemble salient visual attributes and semantic content of natural images, as indicated by the animals’ gaze behavior This suggests the code for object recognition represents compressed features of behavioral relevance, an underexplored aspect of efficient coding. One hypothesis is that the visual system must organize sensory information to build up an “internal model of the environment,” centered around diagnostic motifs of visual objects of “particular key significance for the animal,” as postulated by Horace Barlow[8] These motifs should be related to external features of the visual environment, and to the animal’s actions within it9— for example, related to behaviors such as saccadic eye movements that bring the fovea to salient regions of a scene[10]. We conclude that ventral stream neurons encode information-concentrating features present in the natural visual world, features marked by their relevance to the organism

Objectives

Methods

Results

Conclusion