Statistical basis of nonlinear hebbian learning and application to clustering

Abstract

Recently, the extension of Hebbian learning to nonlinear units has received increased attention. Some successful applications of this learning rule to nonlinear principal component analysis have been reported as well, however, a fundamental understanding of the processing capability of this learning rule in the nonlinear setting is still lacking. In this paper, we pursue a better understanding of what the nonlinear unit is actually doing by exploring the statistical characteristics of the criterion function being optimized and interpreting the operation of the nonlinear activation as a probability integral transformation. To improve the computational capability of the nonlinear units, data preprocessing is suggested. This leads to the development of a two-layer network which consists of linear units in the first layer and nonlinear units in the second layer. The linear units capture and filter the linear aspect (low order correlations) of the data and the nonlinear units discover higher order correlations. Several potential applications are demonstrated through simulated data and previously analyzed data from real measurements. The relationship to exploratory data analysis in statistics is discussed.