Distributed memory and localized memory in neural networks

Abstract

Two major “categories” of unsupervised learning rules are used in artificial neural networks: (i) competitive learning, which is used in the adaptive resonance theory (ART), the self-organizing map, and the neocognitron; and (ii) Hebbian learning without lateral inhibition, which is used in the Hopfield network. Since the competitive learning is essentially Hebbian learning in the presence of lateral inhibition, the author attempts here to discuss general properties of these unsupervised learning rules in a unified paradigm. As a first effort, this paper presents analytical studies of a performance comparison between a competitive learning neural network (CLNN) and the Hopfield neural network (HNN). Specifically, it discusses their abilities as classifiers after they are trained with noisy patterns. First, the HNN is generalized to perform pattern classification in addition to its well-known capability for pattern completion. The Hopfield formulation of the Hebbian learning rule is generalized to allow the existence of noise in training patterns. It is shown that the performance of the generalized HNN as a classifier decreases as noise in training patterns increases. A parallel study is then carried out for a CLNN. First, a simple CLNN is developed with the same components used by the generalized HNN and features used in existing CLNNs. In contrast, this simple CLNN is shown to be robust with respect to noise in training patterns. These discussions suggest that the reason for this difference in performance between the two types of networks is that in the CLNN each synapse is devoted to only one memory, whereas in the HNN each synapse is responsible for many memories. It is concluded that competitive learning, which leads to localized memory, is superior to Hebbian learning without lateral inhibition, which leads to distributed memory, at tolerating noise in training patterns.