Generalization Of Principal Component Analysis Research Articles

Generalized principal component analysis (GPCA)

This paper presents an algebro-geometric solution to the problem of segmenting an unknown number of subspaces of unknown and varying dimensions from sample data points. We represent the subspaces with a set of homogeneous polynomials whose degree is the number of subspaces and whose derivatives at a data point give normal vectors to the subspace passing through the point. When the number of subspaces is known, we show that these polynomials can be estimated linearly from data; hence, subspace segmentation is reduced to classifying one point per subspace. We select these points optimally from the data set by minimizing certain distance function, thus dealing automatically with moderate noise in the data. A basis for the complement of each subspace is then recovered by applying standard PCA to the collection of derivatives (normal vectors). Extensions of GPCA that deal with data in a high-dimensional space and with an unknown number of subspaces are also presented. Our experiments on low-dimensional data show that GPCA outperforms existing algebraic algorithms based on polynomial factorization and provides a good initialization to iterative techniques such as K-subspaces and Expectation Maximization. We also present applications of GPCA to computer vision problems such as face clustering, temporal video segmentation, and 3D motion segmentation from point correspondences in multiple affine views.

Recursive principal components analysis

A recurrent linear network can be trained with Oja's constrained Hebbian learning rule. As a result, the network learns to represent the temporal context associated to its input sequence. The operation performed by the network is a generalization of Principal Components Analysis (PCA) to time-series, called Recursive PCA. The representations learned by the network are adapted to the temporal statistics of the input. Moreover, sequences stored in the network may be retrieved explicitly, in the reverse order of presentation, thus providing a straight-forward neural implementation of a logical stack.

Generalized 2D principal component analysis for face image representation and recognition

In the tasks of image representation, recognition and retrieval, a 2D image is usually transformed into a 1D long vector and modelled as a point in a high-dimensional vector space. This vector-space model brings up much convenience and many advantages. However, it also leads to some problems such as the Curse of Dimensionality dilemma and Small Sample Size problem, and thus produces us a series of challenges, for example, how to deal with the problem of numerical instability in image recognition, how to improve the accuracy and meantime to lower down the computational complexity and storage requirement in image retrieval, and how to enhance the image quality and meanwhile to reduce the transmission time in image transmission, etc. In this paper, these problems are solved, to some extent, by the proposed Generalized 2D Principal Component Analysis (G2DPCA). G2DPCA overcomes the limitations of the recently proposed 2DPCA (Yang et al., 2004) from the following aspects: (1) the essence of 2DPCA is clarified and the theoretical proof why 2DPCA is better than Principal Component Analysis (PCA) is given; (2) 2DPCA often needs much more coefficients than PCA in representing an image. In this work, a Bilateral-projection-based 2DPCA (B2DPCA) is proposed to remedy this drawback; (3) a Kernel-based 2DPCA (K2DPCA) scheme is developed and the relationship between K2DPCA and KPCA (Scholkopf et al., 1998) is explored. Experimental results in face image representation and recognition show the excellent performance of G2DPCA.

A Functional Analysis of Mouse Models of Cardiac Disease through Metabolic Profiling

Since the completion of the human and mouse genomes, the focus in mammalian biology has been on assessing gene function. Tools are needed for assessing the phenotypes of the many mouse models that are now being generated, where genes have been "knocked out," "knocked in," or mutated, so that gene expression can be understood in its biological context. Metabolic profiling of cardiac tissue through high resolution NMR spectroscopy in conjunction with multivariate statistics has been used to classify mouse models of cardiac disease. The data sets included metabolic profiles from mouse models of Duchenne muscular dystrophy, two models of cardiac arrhythmia, and one of cardiac hypertrophy. The metabolic profiles demonstrate that the strain background is an important component of the global metabolic phenotype of a mouse, providing insight into how a given gene deletion may result in very different responses in diverse populations. Despite these differences associated with strain, multivariate statistics were capable of separating each mouse model from its control strain, demonstrating that metabolic profiles could be generated for each disease. Thus, this approach is a rapid method of phenotyping mouse models of disease.

Construction of data-adaptive orthogonal wavelet bases with an extension of principal component analysis

We show that orthonormal bases of functions with multiscale compact supports can be obtained from a generalization of principal component analysis. These functions, called multiscale principal components (MPCs), are eigenvectors of the correlation operator expressed in different vector subspaces. MPCs are data-adaptive functions that minimize their correlation with the reference signal. Using MPCs, we construct orthogonal bases which are similar to dyadic wavelet bases. We observe that MPCs are natural wavelets, i.e. their average is zero or nearly zero if the signal has a dominantly low-pass spectrum. We show that MPCs perform well in simple data compression experiments, in the presence or absence of singularities. We also introduce concentric MPCs, which are orthogonal basis functions having multiscale concentric supports. Use as kernels in convolution products with a signal, these functions allow to define a wavelet transform that has a striking capacity to emphasize atypical patterns.

Principal Geodesic Analysis for the Study of Nonlinear Statistics of Shape

A primary goal of statistical shape analysis is to describe the variability of a population of geometric objects. A standard technique for computing such descriptions is principal component analysis. However, principal component analysis is limited in that it only works for data lying in a Euclidean vector space. While this is certainly sufficient for geometric models that are parameterized by a set of landmarks or a dense collection of boundary points, it does not handle more complex representations of shape. We have been developing representations of geometry based on the medial axis description or m-rep. While the medial representation provides a rich language for variability in terms of bending, twisting, and widening, the medial parameters are not elements of a Euclidean vector space. They are in fact elements of a nonlinear Riemannian symmetric space. In this paper, we develop the method of principal geodesic analysis, a generalization of principal component analysis to the manifold setting. We demonstrate its use in describing the variability of medially-defined anatomical objects. Results of applying this framework on a population of hippocampi in a schizophrenia study are presented.

The Vertical Structure of Wintertime Climate Regimes of the Northern Hemisphere Extratropical Atmosphere

A nonlinear generalization of principal component analysis (PCA), denoted nonlinear principal component analysis (NLPCA), is applied to Northern Hemisphere wintertime geopotential heights at 1000, 700, 500, 300, and 20 hPa. It is found that the optimal nonlinear approximation to the data at the four tropospheric levels is characterized by three equivalent-barotropic regimes of circulation. The NLPCA time series provides a kinematic description of variability within the regimes and transitions between them. The occupation frequencies of the regimes demonstrate substantial interannual and interdecadal variability, some of which can be associated with the influence of El Niño–Southern Oscillation. The stratospheric geopotential height field is also characterized by three circulation regimes. Two of these characterize a vacillation of polar vortex shape and polar temperature. The third regime, which is occupied episodically, is characterized by dramatic reductions in the zonal-mean zonal wind and meridional temperature gradient at high latitudes, and is associated with stratospheric sudden warmings. Evidence is presented for a connection between the stratospheric and tropospheric regimes, including an association between stratospheric sudden warmings and North Atlantic blocks. Finally, these results are compared to previous studies of regime structure and to the Arctic Oscillation paradigm.

Face recognition using independent component analysis and support vector machines

Support vector machines (SVM) and independent component analysis (ICA) are two powerful and relatively recent techniques. SVMs are classifiers which have demonstrated high generalization capabilities in many different tasks, including the object recognition problem. ICA is a feature extraction technique which can be considered a generalization of principal component analysis (PCA). ICA has been mainly used on the problem of blind signal separation. In this paper we combine these two techniques for the face recognition problem. Experiments were made on two different face databases, achieving very high recognition rates. As the results using the combination PCA/SVM were not very far from those obtained with ICA/SVM, our experiments suggest that SVMs are relatively insensitive to the representation space. Thus as the training time for ICA is much larger than that of PCA, this result indicates that the best practical combination is PCA with SVM.

Face recognition by independent component analysis.

A number of current face recognition algorithms use face representations found by unsupervised statistical methods. Typically these methods find a set of basis images and represent faces as a linear combination of those images. Principal component analysis (PCA) is a popular example of such methods. The basis images found by PCA depend only on pairwise relationships between pixels in the image database. In a task such as face recognition, in which important information may be contained in the high-order relationships among pixels, it seems reasonable to expect that better basis images may be found by methods sensitive to these high-order statistics. Independent component analysis (ICA), a generalization of PCA, is one such method. We used a version of ICA derived from the principle of optimal information transfer through sigmoidal neurons. ICA was performed on face images in the FERET database under two different architectures, one which treated the images as random variables and the pixels as outcomes, and a second which treated the pixels as random variables and the images as outcomes. The first architecture found spatially local basis images for the faces. The second architecture produced a factorial face code. Both ICA representations were superior to representations based on PCA for recognizing faces across days and changes in expression. A classifier that combined the two ICA representations gave the best performance.

A generalization of principal component analysis to K sets of variables

The aim of this paper is to introduce a new method, generalized principal component analysis (GPCA), which is a generalization of principal component analysis (PCA), to several data tables. GPCA is a method for both finding common dimensions in several sets of variables and giving a description of each set of variables: GPCA takes into account both the correlation structure within sets and relationships between sets. Two sorts of orthogonal basis are provided; the first basis is useful to represent each set of variables (as PCA does) and the second is useful to represent associations between sets of variables (as canonical correlation analysis does). An example using real data (evolution of five characteristics of car markets from 86 to 93 for 8 countries) illustrates the method.

Sensor and actuator fault isolation by structured partial PCA with nonlinear extensions

Partial PCA based on principal component analysis (PCA) with ideas borrowed from parity relations is a useful method in fault isolation (J. Gertler, W. Li, Y. Huang, T.J. McAvoy, Isolation enhanced principal component analysis, AIChE Journal 45(2) (1999) 323–334). By performing PCA on subsets of variables, a set of structured residuals can be obtained in the same way as structured parity relations. The structured residuals are utilized in composing an isolation scheme for sensor and actuator faults, according to a properly designed incidence matrix. To overcome the limitations of PCA, nonlinear approaches based on generalized PCA (GPCA) and nonlinear PCA (NPCA) are proposed. The nonlinear methods are demonstrated on an artificial 2×2 system while simulation studies on the Tennessee Eastman process illustrate the linear method and some extensions.

Nonlinear Principal Component Analysis by Neural Networks: Theory and Application to the Lorenz System

A nonlinear generalization of principal component analysis (PCA), denoted nonlinear principal component analysis (NLPCA), is implemented in a variational framework using a five-layer autoassociative feed-forward neural network. The method is tested on a dataset sampled from the Lorenz attractor, and it is shown that the NLPCA approximations to the attractor in one and two dimensions, explaining 76% and 99.5% of the variance, respectively, are superior to the corresponding PCA approximations, which respectively explain 60% (mode 1) and 95% (modes 1 and 2) of the variance. It is found that as noise is added to the Lorenz attractor, the NLPCA approximations remain superior to the PCA approximations until the noise level is so great that the lowerdimensional nonlinear structure of the data is no longer manifest to the eye. Finally, directions for future work are presented, and a cinematographic technique to visualize the results of NLPCA is discussed.

The Atmospheric Response over the North Atlantic to Decadal Changes in Sea Surface Temperature

Decadal fluctuations in the climate of the North Atlantic–European region may be influenced by interactions between the atmosphere and the Atlantic Ocean, possibly as part of a coupled ocean–atmosphere mode of variability. For such a mode to exist, a consistent atmospheric response to fluctuations in North Atlantic sea surface temperatures (SST) is required. Furthermore, this response must provide feedbacks to the ocean. Whether a consistent response exists, and whether it yields the required feedbacks, are issues that remain controversial. Here, these issues are addressed using a novel approach to analyze an ensemble of six integrations of the Hadley Centre atmospheric general circulation model HadAM1, all forced with observed global SSTs and sea-ice extents for the period 1949–93. Characterizing the forced atmospheric response is complicated by the presence of internal variability. A generalization of principal component analysis is used to estimate the common forced response given the knowledge of internal variability provided by the ensemble. In the North Atlantic region a remote atmospheric response to El Niño–Southern Oscillation and a further response related to a tripole pattern in North Atlantic SST are identified. The latter, which is most consistent in spring, involves atmospheric circulation changes over the entire region, including a dipole pattern in sea level pressure often associated with the North Atlantic oscillation. Only over the tropical/subtropical Atlantic, however, does it account for a substantial fraction of the total variance. How the atmospheric response could feed back to affect the ocean, and in particular the SST tripole, is investigated. Several potential feedbacks are identified but it has to be concluded that, because of their marginal consistency between ensemble members, a coupled mode that relied on these feedbacks would be susceptible to disruption by internal atmospheric variability. Notwithstanding this conclusion, the authors’ results suggest that predictions of SST evolution could be exploited to predict some aspects of atmospheric variability over the North Atlantic, including fluctuations in spring of the subtropical trade winds and the higher latitude westerlies.

PARAFAC2—Part I. A direct fitting algorithm for the PARAFAC2 model

PARAFAC is a generalization of principal component analysis (PCA) to the situation where a set of data matrices is to be analysed. If each data matrix has the same row and column units, the resulting data are three-way data and can be modelled by the PARAFAC1 model. If each data matrix has the same column units but different (numbers of) row units, the PARAFAC2 model can be used. Like the PARAFACI model, the PARAFAC2 model gives unique solutions under certain mild assumptions, whereas it is less severely constrained than PARAFAC 1. It may therefore also be used for regular three-way data in situations where the PARAFAC1 model is too restricted. Usually the PARAFAC2 model is fitted to a set of matrices with cross-products between the column units. However, this model-fitting procedure is computationally complex and inefficient. In the present paper a procedure for fitting the PARAFAC2 model directly to the set of data matrices is proposed. It is shown that this algorithm is more efficient than the indirect fitting algorithm. Moreover, it is more easily adjusted so as to allow for constraints on the parameter matrices, to handle missing data, as well as to handle generalizations to sets of three- and higher-way data. Furthermore, with the direct fitting approach we also gain information on the row units, in the form of 'factor scores'. As will be shown, this elaboration of the model in no way limits the feasibility of the method. Even though full information on the row units becomes available, the algorithm is based on the usually much smaller cross-product matrices only. Copyright (C) 1999 john Wiley & Sons, Ltd.

PARAFAC2 - Part I. A direct fitting algorithm for the PARAFAC2 model

PARAFAC is a generalization of principal component analysis (PCA) to the situation where a set of data matrices is to be analysed. If each data matrix has the same row and column units, the resulting data are three-way data and can be modelled by the PARAFAC1 model. If each data matrix has the same column units but different (numbers of) row units, the PARAFAC2 model can be used. Like the PARAFAC1 model, the PARAFAC2 model gives unique solutions under certain mild assumptions, whereas it is less severely constrained than PARAFAC1. It may therefore also be used for regular three-way data in situations where the PARAFAC1 model is too restricted. Usually the PARAFAC2 model is fitted to a set of matrices with cross-products between the column units. However, this model-fitting procedure is computationally complex and inefficient. In the present paper a procedure for fitting the PARAFAC2 model directly to the set of data matrices is proposed. It is shown that this algorithm is more efficient than the indirect fitting algorithm. Moreover, it is more easily adjusted so as to allow for constraints on the parameter matrices, to handle missing data, as well as to handle generalizations to sets of three- and higher-way data. Furthermore, with the direct fitting approach we also gain information on the row units, in the form of ‘factor scores’. As will be shown, this elaboration of the model in no way limits the feasibility of the method. Even though full information on the row units becomes available, the algorithm is based on the usually much smaller cross-product matrices only. Copyright © 1999 John Wiley & Sons, Ltd.

Metric-based principal components:. data uncertainties

Seeking an index characterizing the best-determined mode of variability leads to a natural generalization of principal-component analysis with an explicit metric characterizing the uncertainties of the data. This formalism, which distinguishes between state-space patterns and patterns of coefficients defining principal components, allows the more accurate data to exert a greater influence on the definition of the indices than they do in conventional principal-component analysis; in all other aspects, the new formalism is the same as the old. Within the context of the simple example of Bretherton and collaborators, metric-based principal-component analysis is shown to be capable of finding correlated patterns of variability in two different data sets.

An Information-Maximization Approach to Blind Separation and Blind Deconvolution

We derive a new self-organizing learning algorithm that maximizes the information transferred in a network of nonlinear units. The algorithm does not assume any knowledge of the input distributions, and is defined here for the zero-noise limit. Under these conditions, information maximization has extra properties not found in the linear case (Linsker 1989). The nonlinearities in the transfer function are able to pick up higher-order moments of the input distributions and perform something akin to true redundancy reduction between units in the output representation. This enables the network to separate statistically independent components in the inputs: a higher-order generalization of principal components analysis. We apply the network to the source separation (or cocktail party) problem, successfully separating unknown mixtures of up to 10 speakers. We also show that a variant on the network architecture is able to perform blind deconvolution (cancellation of unknown echoes and reverberation in a speech signal). Finally, we derive dependencies of information transfer on time delays. We suggest that information maximization provides a unifying framework for problems in "blind" signal processing.

Optimization of an observation well network for monitoring groundwater level using statistical methods : M. Hussain, K. Fuchs & J. Fank, in: Groundwater quality: remediation and protection. Proc. conference, Prague, 1995, ed K. Kovar & J. Krasny, (IAHS; Publication, 225), 1995, pp 211–219

Abstract The purpose of this paper is to optimize a network of observation wells in southeast of Austria, developed for monitoring groundwater level and groundwater resource in this area. As groundwater level data are collected over time, problems arise in defining an overall variogram valid for the whole period of observation, which is needed to estimate groundwater level properly. This paper compares three approaches - arithmetic mean of every series, general relative variogram and principal component analysis - to calculate an overall variogram and discusses the differences between the approaches in optimizing the network for the above mentioned area. Universal kriging was then used to map the minimum error variance estimates for each of these three approaches. General relative variogram appeared to give the best results. THEORETICAL BACKGROUND The variogram plays a central role in optimizing observation well network for monitoring groundwater level. A valid variogram model has to be selected, and the parameters, of that model should be estimated before kriging (spatial prediction) can be performed. These inference procedures are generally based on examination of empirical variograms, which describe the spatial dispersion in the area of interest (see Clark, 1979). As the groundwater level data are collected over time, applying such space oriented approaches to spatiotemporal process may lead to loss of valuable information in the time dimension. Problem arises in defining an overall variogram valid for the whole period of observation. Consider two numerical values z(x) and z(x + h), at two points x and x + h separated by the vector h. The variability between these two quantities is characterized by the variogram function 2y(x,h), which is defined as the expectation of the random variable 2y(x,h) = E{[Z(x) - Z(x + h)]

Generalizations of principal component analysis, optimization problems, and neural networks

We derive and discuss various generalizations of neural PCA (Principal Component Analysis)-type learning algorithms containing nonlinearities using optimization-based approach. Standard PCA arises as an optimal solution to several different information representation problems. We justify that this is essentially due to the fact that the solution is based on the second-order statistics only. If the respective optimization problems are generalized for nonquadratic criteria so that higher-order statistics are taken into account, their solutions will in general be different. The solutions define in a natural way several meaningful extensions of PCA and give a solid foundation for them. In this framework, we study more closely generalizations of the problems of variance maximization and mean-square error minimization. For these problems, we derive gradient-type neural learning algorithms both for symmetric and hierarchic PCA-type networks. As an important special case, the well-known Sanger's generalized Hebbian algorithm (GHA) is shown to emerge from natural optimization problems.

Adaptive Principal Surfaces

Abstract We develop a nonlinear generalization of principal components analysis. A principal surface of the data is constructed adaptively, using some ideas from the MARS procedure of Friedman. We explore applications to curve and surface reconstruction and to data summarization.