Internal Representations of Sensory Input Reflect the Motor Output with Which Organisms Respond to the Input

Abstract

What determines how sensory input is internally represented? The traditional answer is that internal representations of sensory input reflect the properties of the input. This answer is based on a passive or contemplative view of our knowledge of the world which is rooted in the philosophical tradition and, in psychology, appears to be almost mandatory given the fact that, in laboratory experiments, it is much easier for the researcher to control and manipulate the sensory input which is presented to the experimental subjects than the motor output with which the subjects respond to the input. However, a minority view which is gaining increasing support (Gibson, 1986; O’Regan and Noe, in press) is that internal representations are instead action-based, that is, that the manner in which organisms internally represent the sensory input reflects the properties of the actions with which the organisms respond to the sensory input rather than the properties of the sensory input. In this chapter we describe a series of computer simulations using neural networks that tend to support the action-based view of internal representations. Internal representations in neural networks are not symbolic or semantic entities, like cognitivist representations (Fodor, 1981), but they are patterns of activation states in the network’s internal units which are caused by input activation patterns and which in turn cause activation patterns in the network’s output units. Our networks are sensory-motor neural networks. Their input units encode sensory input and their output units encode changes in the physical location of the organism’s body or body parts, i.e., movements. We train networks to execute a number of sensory-motor tasks and by examining their internal representations at the end of training we determine whether these internal representations co-vary with the properties of the sensory input or with the properties of the motor output. The chapter describes three sets of simulations. In the first set we distinguish between micro-actions and macro-actions and we show that both micro-actions and macro-actions are real for neural networks. Micro-actions are the successive movements that make up an entire goal-directed action, and each micro-action is encoded in the activation pattern observed in the network’s motor output units in a single input/output cycle. Macro-actions are sequences of micro-actions that allow the organism to reach some goal. Our simulations show that internal representations encode, i.e., reflect the properties of, macro-actions. In the second set of simulations we show that if there is a succession of layers of internal units from the sensory input to the motor output the layers which are closer to the sensory input will tend to reflect the properties of the input and those closer to the motor output the properties of the output. However, in the third and final set of simulations we also show that the actions with which the organism responds to the input dictate the form of internal representations as low down the succession of in