Multimodal System Based on Self-organizing Maps

Abstract

We present experiments with a multimodal system based on a novel variant of the Self-Organizing Map (SOM) called the Associative Self-Organizing Map (A-SOM). The A-SOM is similar to the SOM and develops a representation of its input space, but also learns to associate its activity with additional inputs, e.g. the activities of one or several external SOMs. This enables the modelling of expectations in one sensory modality due to the activity elicited in another modality. The paper presents the A-SOM algorithm generalized to an arbitrary number of (possibly delayed) associated activities together with simulation results with a multimodal sensory system and its extension to a system that also includes an action network. The simulation results were very encouraging and confirmed: The ability of the A-SOM to learn to associate the representations of its input space with the representations of the input spaces developed in two connected SOMs; The ability of the extended system to elicit proper activity in the action network; The simulations demonstrated good generalization ability.