Robust hebbian learning and noisy principal component analysis

Abstract

The classical analysis of a stochastic signal into principal components compresses the signal using an optimal selection of linear features. Noisy Principal Component Analysis (NPCA) is an extension of PCA under the assumption that the extracted features are unreliable, and the unreliability is modeled by additive noise. The applications of this assumption appear for instance, in communications problems with noisy channels. The level of noise in the NPCA features affects the reconstruction error in a way resembling the water-filling analogy in information theory. Robust neural network models for Noisy PCA can be defined with respect to certain synaptic weight constraints. In this paper we present the NPCA theory related to a particularly simple and tractable constraint which allows us to evaluate the robustness of old PCA Hebbian learning rules. It turns out that those algorithms are not optimally robust in the sense that they produce a zero solution when the noise power level reaches half the limit set by NPCA. In fact, they are not NPCA-optimal for any other noise levels except zero. Finally, we propose new NPCA-optimal robust Hebbian learning algorithms for multiple adaptive noisy principal component extraction.