Specialization processes in on-line unsupervised learning

Abstract

Abstract From the recent analysis of supervised learning by on-line gradient descent in multilayered neural networks it is known that the necessary process of student specialization can be delayed significantly. We demonstrate that this phenomenon also occurs in various models of unsupervised learning. A solvable model of competitive learning is presented, which identifies prototype vectors suitable for the representation of high-dimensional data. The specific case of two overlapping clusters of data and a matching number of prototype vectors exhibits non-trivial behaviour like almost stationary plateau configurations. As a second example scenario we investigate the application of Sanger's algorithm for principal component analysis in the presence of two relevant directions in input space. Here, the fast learning of the first principal component may lead to an almost complete loss of initial knowledge about the second one.