Norms are Not the Norm: Testing Theories of Sensory Encoding Using Visual Aftereffects

Abstract

The goal of visual neuroscience is to explain how patterns of neural activity give rise to visual experiences. Here I use visual aftereffects to explore the computational principles underlying the perception of simple, intermediate, and complex forms (orientations, shapes, and faces). Aftereffects occur when exposure to one stimulus changes the appearance of a subsequent stimulus. In the tilt aftereffect, for example, staring at a left-tilted line can make a vertical line seem briefly to lean to the right. Aftereffects are thought to be caused by neural adaptation --- changes in the responsiveness of neurons after prolonged stimulation. Visual aftereffects allow us to probe sensory encoding, because adaptation within different encoding schemes can predict different patterns of perceptual aftereffect. I focus on two theories of encoding prominent in the face recognition literature. According to norm-based accounts, faces are represented in terms of how they deviate from a unique norm, such as the average of all experienced faces. Similar proposals have been raised in other contexts, for example to encode the aspect ratio of shapes. Norm-based encoding can be implemented by a population in which neurons respond increasingly as a stimulus moves further in the neuron's preferred direction, away from a normative value in stimulus space. Alternatively, exemplar-based accounts propose that faces are encoded in terms of their similarity to a number of previously-experienced exemplars. Exemplar-based encoding can be implemented by a population of non-monotonically tuned neurons that prefer particular points in stimulus space, rather than directions. Norm-based and exemplar-based accounts make distinct predictions for the pattern of aftereffects one should experience following adaptation. Norm-based encoding is associated with renormalisation, in which the adapting stimulus appears more 'neutral' or 'average' after prolonged viewing, and the appearances of other stimuli are altered in the same fashion (e.g. after adapting to a male face, all faces should look more feminine). Exemplar-based encoding is associated with 'local repulsion,' in which the adapting stimulus appears unchanged, but differences between it and subsequent stimuli are exaggerated (e.g. after adapting to a male face, a very masculine face should look even more masculine, while an androgynous or female face should look more feminine). Over the past fifteen years, norm-based theories of shape and face encoding have gained traction, ostensibly supported by evidence from aftereffects. In Chapter 1, I review this evidence, and relate it to an older debate concerning the role of norms in orientation perception. I conclude that the evidence for renormalisation in form perception is underwhelming. In Chapter 2 I empirically address the most recent evidence for orientation renormalisation. I show that these data likely arise from an interaction between the task used, and observers' uneven orientation sensitivity. I therefore conclude that orientation aftereffects are best described as purely locally-repulsive. In order to draw inferences about face and shape representation from aftereffects, it is important to exclude the possibility that aftereffects between complex stimuli are caused entirely by adaptation to lower-level image components, such as local edge orientations. In Chapter 3 I show that shape aspect-ratio aftereffects are determined more strongly by perceived shape than retinal shape, and therefore involve computations relatively late in the visual processing hierarchy. In Chapter 4 I show that this higher-level component manifests as a locally-repulsive aftereffect, consistent with an exemplar-based code for aspect ratio. Having established that tilt and shape aftereffects are more consistent with exemplar-based than norm-based representation, I return my attention to face aftereffects. In Chapter 5 I devise a more diagnostic test than those previously used, by substituting the common binary-classification task for a ternary-classification task. I find that facial gender aftereffects are consistent with local repulsion, while facial distortion aftereffects do not neatly match the predictions of either theory. This raises the previously unsuspected possibility that adaptation along different facial dimensions might involve distinct patterns of perceptual changes. In Chapter 6 I extend this work by devising a second task, which is more flexible than the first and minimises potential response biases. In a series of experiments I compare appearance between faces presented in differently-adapted retinal conditions. Data show that both facial gender and facial identity aftereffects are consistent with local repulsion, but not renormalisation. Despite the recent popularity enjoyed by norm-based theories of shape and face encoding, I find that shape and face aftereffects manifest almost exclusively as local repulsions away from the adapted value, consistent with adaptation within an exemplar-based code. In Chapter 7 I discuss these results, and consider their implications for computational models of sensory encoding. The consistency of my results across simple, intermediate, and complex stimuli suggests that the brain may use similar exemplar-based encoding strategies throughout spatial vision.