Visual memories and mental images

Abstract

A model is presented for the architecture of the neural networks which encode visual information for storage and which reconstruct iconic representations from storage representations. (Iconic representations are geometrically similar to projections of the objects they represent.) Each storage representation consists of a sequence of patterns derived while the eyes fixate at different positions in the visual field. Each pattern in the sequence has three components: (1) a control component which describes both where the eyes fixated and the size of the attended scene fragment; (2) a surface quality component which describes visual surface characteristics of the object; and (3) a spatial component which describes the spatial extent, spatial position (depth), surface orientation and visual flow (movement) of the surface having the specified surface characteristics. Prior to storage, all spatial components are transformed using a complex logarithmic mapping. As a consequence, stored spatial patterns are not iconic representations of the scene fragments they represent. Also, storage representations can be recognized and reconstructed at any desired size and orientation: they are size and orientation invariant. During reconstruction, each pattern in the storage representation is transformed back into an iconic representation using a complex exponential mapping. One consequence of the combined complex logarithmic and exponential mappings and the limited size of the storage representations is that the fidelity of the recalled information degrades exponentially from its centre. A neural network, called spatial memory, not only holds the partially reconstructed representation during recall, but also shifts it to remain in registration with the fragment currently being recalled and combined. The control system uses the control component of each stored pattern and knowledge of the size and orientation of the reconstruction to determine how to shift the partially reconstructed representation in spatial memory. Due to the decreasing fidelity from the centre to the perimeter of each reconstructed scene fragment, spatial memory only preserves information from overlapping fragments having the highest fidelity. It does so by maintaining and using fidelity information for each position in the reconstructed representation. Spatial memory can maintain a current stable representation of the visual world. It can also magnify, reduce, shift and rotate representations. The representations are therefore independent of their position in spatial memory. It is suggested that the representations held and processed by spatial memory correspond to the representations we call mental images and for this reason they are called mental images in the model.