Principal Component Analysis Learning Research Articles

A ROBUST AND GLOBALLY CONVERGENT PCA LEARNING ALGORITHM

Principal component analysis (PCA) using neural networks is an active research field with many applications to signal processing and data analysis. This paper presents a PCA neural network endowed with a novel learning algorithm, and an analysis of its features. As the basic discrete-time Oja's PCA neural network does not converge globally, it is important to derive a robust and globally convergent PCA learning algorithm. Based on previous works on the globally convergent PCA learning algorithm, a robust and globally convergent PCA learning algorithm is proposed in this paper. The behavior of this discrete-time learning algorithm is directly studied in this paper. We show that the algorithm is robust and globally convergent. The selection of the parameters of this algorithm will also be discussed in details. Finally, simulation results are provided to verify the theoretical results presented. Compared to other PCA learning algorithms, the proposed algorithm performs favorably in terms of robust stability, global convergence, speed and accuracy.

Global convergence of Oja's PCA learning algorithm with a non-zero-approaching adaptive learning rate

A non-zero-approaching adaptive learning rate is proposed to guarantee the global convergence of Oja's principal component analysis (PCA) learning algorithm. Most of the existing adaptive learning rates for Oja's PCA learning algorithm are required to approach zero as the learning step increases. However, this is not practical in many applications due to the computational round-off limitations and tracking requirements. The proposed adaptive learning rate overcomes this shortcoming. The learning rate converges to a positive constant, thus it increases the evolution rate as the learning step increases. This is different from learning rates which approach zero which slow the convergence considerably and increasingly with time. Rigorous mathematical proofs for global convergence of Oja's algorithm with the proposed learning rate are given in detail via studying the convergence of an equivalent deterministic discrete time (DDT) system. Extensive simulations are carried out to illustrate and verify the theory derived. Simulation results show that this adaptive learning rate is more suitable for Oja's PCA algorithm to be used in an online learning situation.

Convergence Analysis of a Deterministic Discrete Time System of Oja's PCA Learning Algorithm

The convergence of Oja's principal component analysis (PCA) learning algorithms is a difficult topic for direct study and analysis. Traditionally, the convergence of these algorithms is indirectly analyzed via certain deterministic continuous time (DCT) systems. Such a method will require the learning rate to converge to zero, which is not a reasonable requirement to impose in many practical applications. Recently, deterministic discrete time (DDT) systems have been proposed instead to indirectly interpret the dynamics of the learning algorithms. Unlike DCT systems, DDT systems allow learning rates to be constant (which can be a nonzero). This paper will provide some important results relating to the convergence of a DDT system of Oja's PCA learning algorithm. It has the following contributions: 1) A number of invariant sets are obtained, based on which we can show that any trajectory starting from a point in the invariant set will remain in the set forever. Thus, the nondivergence of the trajectories is guaranteed. 2) The convergence of the DDT system is analyzed rigorously. It is proven, in the paper, that almost all trajectories of the system starting from points in an invariant set will converge exponentially to the unit eigenvector associated with the largest eigenvalue of the correlation matrix. In addition, exponential convergence rate are obtained, providing useful guidelines for the selection of fast convergence learning rate. 3) Since the trajectories may diverge, the careful choice of initial vectors is an important issue. This paper suggests to use the domain of unit hyper sphere as initial vectors to guarantee convergence. 4) Simulation results will be furnished to illustrate the theoretical results achieved.

A globally convergent learning algorithm for PCA neural networks

Principal component analysis (PCA) by neural networks is one of the most frequently used feature extracting methods. To process huge data sets, many learning algorithms based on neural networks for PCA have been proposed. However, traditional algorithms are not globally convergent. In this paper, a new PCA learning algorithm based on cascade recursive least square (CRLS) neural network is proposed. This algorithm can guarantee the network weight vector converges to an eigenvector associated with the largest eigenvalue of the input covariance matrix globally. A rigorous mathematical proof is given. Simulation results show the effectiveness of the algorithm.

TIME-ORIENTED HIERARCHICAL METHOD FOR COMPUTATION OF PRINCIPAL COMPONENTS USING SUBSPACE LEARNING ALGORITHM

Principal Component Analysis (PCA) and Principal Subspace Analysis (PSA) are classic techniques in statistical data analysis, feature extraction and data compression. Given a set of multivariate measurements, PCA and PSA provide a smaller set of "basis vectors" with less redundancy, and a subspace spanned by them, respectively. Artificial neurons and neural networks have been shown to perform PSA and PCA when gradient ascent (descent) learning rules are used, which is related to the constrained maximization (minimization) of statistical objective functions. Due to their low complexity, such algorithms and their implementation in neural networks are potentially useful in cases of tracking slow changes of correlations in the input data or in updating eigenvectors with new samples. In this paper we propose PCA learning algorithm that is fully homogeneous with respect to neurons. The algorithm is obtained by modification of one of the most famous PSA learning algorithms--Subspace Learning Algorithm (SLA). Modification of the algorithm is based on Time-Oriented Hierarchical Method (TOHM). The method uses two distinct time scales. On a faster time scale PSA algorithm is responsible for the "behavior" of all output neurons. On a slower scale, output neurons will compete for fulfillment of their "own interests". On this scale, basis vectors in the principal subspace are rotated toward the principal eigenvectors. At the end of the paper it will be briefly analyzed how (or why) time-oriented hierarchical method can be used for transformation of any of the existing neural network PSA method, into PCA method.

A class of learning algorithms for principal component analysis and minor component analysis

Principal component analysis (PCA) and minor component analysis (MCA) are a powerful methodology for a wide variety of applications such as pattern recognition and signal processing. In this paper, we first propose a differential equation for the generalized eigenvalue problem.We prove that the stable points of this differential equation are the eigenvectors corresponding to the largest eigenvalue. Based on this generalized differential equation, a class of PCA and MCA learning algorithms can be obtained. We demonstrate that many existing PCA and MCA learning algorithms are special cases of this class, and this class includes some new and simpler MCA learning algorithms. Our results show that all the learning algorithms of this class have the same order of convergence speed, and they are robust to implementation error.

A class of learning algorithms for principal component analysis and minor component analysis

Principal component analysis (PCA) and minor component analysis (MCA) are a powerful methodology for a wide variety of applications such as pattern recognition and signal processing. In this paper, we first propose a differential equation for the generalized eigenvalue problem.We prove that the stable points of this differential equation are the eigenvectors corresponding to the largest eigenvalue. Based on this generalized differential equation, a class of PCA and MCA learning algorithms can be obtained. We demonstrate that many existing PCA and MCA learning algorithms are special cases of this class, and this class includes some new and simpler MCA learning algorithms. Our results show that all the learning algorithms of this class have the same order of convergence speed, and they are robust to implementation error.

A general theory of a class of linear neural nets for principal and minor component analysis

This paper presents a unified theory of a class of learning neural nets for principal component analysis (PCA) and minor component analysis (MCA). First, some fundamental properties are addressed which all neural nets in the class have in common. Second, a subclass called the generalized asymmetric learning algorithm is investigated, and the kind of asymmetric structure which is required in general to obtain the individual eigenvectors of the correlation matrix of a data sequence is clarified. Third, focusing on a single-neuron model, a systematic way of deriving both PCA and MCA learning algorithms is shown, through which a relation between the normalization in PCA algorithms and that in MCA algorithms is revealed.

The nonlinear PCA criterion in blind source separation: Relations with other approaches

We present new results on the nonlinear principal component analysis (PCA) criterion in blind source separation (BSS). We derive the criterion in a form that allows easy comparisons with other BSS and independent component analysis (ICA) contrast functions like cumulants, Bussgang criteria, and information theoretic contrasts. This clarifies how the nonlinearity should be chosen optimally. We also discuss the connections of the nonlinear PCA learning rule with the Bell-Sejnowski algorithm and the adaptive EASI algorithm. Furthermore, we show that a nonlinear PCA criterion can be minimized using least-squares approaches, leading to computationally efficient and fast converging algorithms. The paper shows that nonlinear PCA is a versatile starting point for deriving different kinds of algorithms for blind signal processing problems.

Handwritten Digit Recognition by a Mixture of Local PrincipalComponent Analysis

Mixture of local principal component analysis (PCA) has attracted attention due to a number of benefits over global PCA. The performance of a mixture model usually depends on the data partition and local linear fitting. In this paper, we propose a mixture model which has the properties of optimal data partition and robust local fitting. Data partition is realized by a soft competition algorithm called neural ’gas‘ and robust local linear fitting is approached by a nonlinear extension of PCA learning algorithm. Based on this mixture model, we describe a modular classification scheme for handwritten digit recognition, in which each module or network models the manifold of one of ten digit classes. Experiments demonstrate a very high recognition rate.

The nonlinear PCA learning rule in independent component analysis

It has been verified experimentally that when nonlinear Principal Component Analysis (PCA) learning rules are used for the weights of a neural layer, the neurons have signal separation capabilities. The network is performing Independent Component Analysis. The learning rule, earlier proposed by the author, is studied here mathematically to analyze why and how the algorithm works in this application. It is shown that the weight matrix obtained as the asymptotic solution of the nonlinear PCA learning rule is in some cases a rotation of the input vector to statistically independent directions. This explains why it can be used for image and speech signal separation. Sufficient conditions are formulated, depending on the nonlinear neuron activation function and on the probability densities of the original signal components. It is shown that a sigmoidal nonlinearity as the activation function is feasible for flat sub-Gaussian densities of the original signals, while polynomial activation functions are feasible for sharp super-Gaussian densities.

Effect of Binocular Cortical Misalignment on Ocular Dominance and Orientation Selectivity

We model a two-eye visual environment composed of natural images and study its effect on single cell synaptic modification. In particular, we study the effect of binocular cortical misalignment on receptive field formation after eye opening. We show that binocular misalignment affects principal component analysis (PCA) and Bienenstock, Cooper, and Munro (BCM) learning in different ways. For the BCM learning rule this misalignment is sufficient to produce varying degrees of ocular dominance, whereas for PCA learning binocular neurons emerge in every case.

Robust principal component analysis by self-organizing rules based on statistical physics approach

This paper applies statistical physics to the problem of robust principal component analysis (PCA). The commonly used PCA learning rules are first related to energy functions. These functions are generalized by adding a binary decision field with a given prior distribution so that outliers in the data are dealt with explicitly in order to make PCA robust. Each of the generalized energy functions is then used to define a Gibbs distribution from which a marginal distribution is obtained by summing over the binary decision field. The marginal distribution defines an effective energy function, from which self-organizing rules have been developed for robust PCA. Under the presence of outliers, both the standard PCA methods and the existing self-organizing PCA rules studied in the literature of neural networks perform quite poorly. By contrast, the robust rules proposed here resist outliers well and perform excellently for fulfilling various PCA-like tasks such as obtaining the first principal component vector, the first k principal component vectors, and directly finding the subspace spanned by the first k vector principal component vectors without solving for each vector individually. Comparative experiments have been made, and the results show that the authors' robust rules improve the performances of the existing PCA algorithms significantly when outliers are present.

Influence Function Analysis of PCA and BCM Learning

Based on the influence function (Hampel et al. 1986), we analyze several forms of learning rules under the influence of abnormal inputs (outliers). Principal component analysis (PCA) learning rules (Oja 1982, 1989, 1992) are shown to be sensitive to the influence of outliers. Biologically motivated robust PCA learning rules are proposed. The variants of BCM learning (Bienenstock et al. 1982; Intrator 1990b) with linear neurons are shown to be subject to the influence of outliers, in contrast they are shown to be robust to outliers with sigmoidal neurons.

Principal component analysis learning algorithms: a neurobiological analysis

The biological relevance of principal component analysis (PCA) learning algorithms is addressed by: (i) describing a plausible biological mechanism which accounts for the changes in synaptic efficacy implicit in Oja's 'Subspace' algorithm (Int. J. neural Syst. 1, 61 (1989)); and (ii) establishing a potential role for PCA-like mechanisms in the development of functional segregation. PCA learning algorithms comprise an associative Hebbian term and a decay term which interact to find the principal patterns of correlations in the inputs shared by a group of units. We propose that the presynaptic component of this decay could be regulated by retrograde signals that are translocated from the terminal arbors of presynaptic neurons to their cell bodies. This proposal is based on reported studies of structural plasticity in the nervous system. By using simulations we demonstrate that PCA-like mechanisms can eliminate afferent connections whose signals are unrelated to the prevalent pattern of afferent activity. This elimination may be instrumental in refining extrinsic cortico-cortical connections that underlie functional segregation.