Figure-ground representation in deep neural networks

Abstract

Deep neural networks achieve state-of-the-art performance on many image segmentation tasks. However, the nature of the learned representations used by these networks is unclear. Biological brains solve this task very efficiently and seemingly effortlessly. Neurophysiological recordings have begun to elucidate the underlying neural mechanisms of image segmentation. In particular, it has been proposed that border ownership selectivity (BOS) is the first step in this process in the brain. BOS is a property of an orientation selective neuron to differentially respond to an object contour dependent on the location of the foreground object (figure). We explored whether deep neural networks use representations close to those of biological brains, in particular whether they explicitly represent BOS. We therefore developed a suite of in-silico experiments to test for BOS, similar to experiments that have been used to probe primate BOS. We tested two deep neural networks trained for scene segmentation tasks (DOC [1] and Mask R-CNN [2]), as well as one network trained for object recognition (ResNet-50 [3]). Units in ResNet50 predominantly showed contrast tuning. Units in Mask R-CNN responded weakly to the test stimuli. In the DOC network, we found that units in earlier layers of the network showed stronger contrast tuning, while units in deeper layers of the network showed increasing BOS. In primate brains, contrast tuning seems wide-spread in extrastriate areas while BOS is most common in intermediate area V2 where the prevalence of BOS neurons exceeds that of earlier (V1) and later (V4) areas. We also found that the DOC network, which was trained on natural images, did not generalize well to the simple stimuli typically used in experiments. This differs from findings in biological brains where responses to simple stimuli are stronger than to complex natural scenes. Our methods are general and can also be applied to other deep neural networks and tasks.