Projection Pursuit Algorithm Research Articles

The Convallis learning rule for unsupervised learning in spiking neuronal networks

Learning is often considered to be governed by reinforcement, causing animals to increase the probability of performing rewarded behaviours, and decrease the probability of behaviours leading to harmful outcomes. However, not all learning is governed by reward. Mere exposure to a novel environment or set of sensory stimuli, unpaired with any behaviour or reinforcement, leads to perceptual learning that allows the animal to more readily form behavioural associations with these stimuli, should the need later arise. In statistics and machine learning, the problem of forming representations of a data set without any explicit training is called learning. Artificial neural networks that perform unsupervised learning have been described, primarily for the case of firing rate neurons. Many of these learning rules allow the networks to perform analyses equivalent to standard statistical techniques. For example, a variant of the Bienenstock, Cooper and Monro (BCM) theory of plasticity allows neurons to implement projection pursuit, a statistical technique for searching for non-Gaussian projections of input data. Here we describe a synaptic rule for unsupervised learning in spiking neurons. The rule is derived as gradient optimization of the time integral of a nonlinear function of membrane potential. This function is shaped like a valley, leading to a preference for membrane potentials near rest or spike threshold, but avoiding intermediate-level potentials (the name comes from the Latin for valley). To avoid saturation, the rule is stabilized by a homeostatic mechanism that enforces a constant firing rate using a PI controller that scales synaptic weights. This combination causes neurons to develop a skewed, non-Gaussian distribution of membrane potentials analogously to the projection pursuit algorithm. Using the TIDIGITS database of spoken digit utterances, we show the rule allows a recurrent network of spiking excitatory and inhibitory neurons to develop selective representations of these digits. Applying a linear classifier downstream of the recurrent network shows the Convallis rule allows substantially better readout than a purely random network (liquid-state machine), the homeostatic rule alone, or various other plasticity rules for spiking neurons. Applying the rule to simulations of in vitro plasticity paradigms, we find that it reproduces several published results including STDP, although STDP alone cannot produce similar performance in speech recognition. We suggest that ability to perform real-world information processing tasks provides a useful way to constrain theories of synaptic plasticity.

Study on Gold Price Forecasting Technique Based on Neural Network Optimized by GA with Projection Pursuit Algorithm

Gold price has significant nonlinearity and time-variance with many indeterminate influencing factors. In order to improve the forecast accuracy of gold price, this paper puts forward a gold price forecast model combing projection pursuit with neural network. At first, projection pursuit algorithm is used to screen the influencing factors, and then the influencing factors are used as the input variables of BP neural network to learn. Meanwhile, this paper applies genetic algorithm to optimize BP neural network and build gold price forecast model. At last, the forecast performance of the model is tested through simulation experiment. Experimental results show that the combined model can well describe the variation trend of gold price, simplify the network structure and speed up network convergence, which effectively improves the forecast accuracy and operating efficiency of gold price and provides a new forecast method for gold price.

A Credit Rating Model for Enterprises Based on Projection Pursuit and K-Means Clustering Algorithm

This paper proposes a new credit rating model for enterprises based on Projection Pursuit and K-means clustering algorithm. Firstly, using Projection Pursuit, the comprehensive credit score of each sample is obtained, so as to reflect the structure or characteristics of original multi-dimensional data. Secondly, the distribution density of the comprehensive credit score series is estimated by the kernel density estimation method, and then the initial cluster centers in original high dimension space are determined according to the local maximum points of density function. Finally, starting from the initial cluster centers above, using K-means clustering algorithm, the final cluster centers are obtained, and then the credit grades are partitioned. Thus, the credit rating for enterprises is realized. Taking the high-tech listed companies in China as samples, it is proved that the model proposed by this paper is feasible and effective.

Comprehensive Evaluation Method for Interchange Project Based on Projection Pursuit

Comprehensive evaluation method for interchange project is built on the basic principle and the solution process of the projection pursuit algorithm. The method can determine directly the objective weights of the indexes from the project of being evaluated, and avoid the subjective negative impact on the conclusion of the determining index weights and the conclusion influenced by the index weights given by people themselves in other evaluation methods. Real coding based Accelerating Genetic Algorithm (RAGA) is proposed and applied to deal with global optimization problem of high-dimensional data effectively by optimizing the projection index function and searching for the optimal projection directional vector.

Iterative Gaussianization: From ICA to Random Rotations

Most signal processing problems involve the challenging task of multidimensional probability density function (PDF) estimation. In this paper, we propose a solution to this problem by using a family of rotation-based iterative Gaussianization (RBIG) transforms. The general framework consists of the sequential application of a univariate marginal Gaussianization transform followed by an orthonormal transform. The proposed procedure looks for differentiable transforms to a known PDF so that the unknown PDF can be estimated at any point of the original domain. In particular, we aim at a zero-mean unit-covariance Gaussian for convenience. RBIG is formally similar to classical iterative projection pursuit algorithms. However, we show that, unlike in PP methods, the particular class of rotations used has no special qualitative relevance in this context, since looking for interestingness is not a critical issue for PDF estimation. The key difference is that our approach focuses on the univariate part (marginal Gaussianization) of the problem rather than on the multivariate part (rotation). This difference implies that one may select the most convenient rotation suited to each practical application. The differentiability, invertibility, and convergence of RBIG are theoretically and experimentally analyzed. Relation to other methods, such as radial Gaussianization, one-class support vector domain description, and deep neural networks is also pointed out. The practical performance of RBIG is successfully illustrated in a number of multidimensional problems such as image synthesis, classification, denoising, and multi-information estimation.

An adaptive wavelet neural network for spatio-temporal system identification

Starting from the basic concept of coupled map lattices, a new family of adaptive wavelet neural networks (AWNN) is introduced for spatio-temporal system identification, by combining an efficient wavelet representation with a coupled map lattice model. A new orthogonal projection pursuit (OPP) method, coupled with a particle swarm optimization (PSO) algorithm, is proposed for augmenting the proposed network. A novel two-stage hybrid training scheme is developed for constructing a parsimonious network model. In the first stage, by applying the orthogonal projection pursuit algorithm, significant wavelet neurons are adaptively and successively recruited into the network, where adjustable parameters of the associated wavelet neurons are optimized using a particle swarm optimizer. The resultant network model, obtained in the first stage, may however be redundant. In the second stage, an orthogonal least squares algorithm is then applied to refine and improve the initially trained network by removing redundant wavelet neurons from the network. The proposed two-stage hybrid training procedure can generally produce a parsimonious network model, where a ranked list of wavelet neurons, according to the capability of each neuron to represent the total variance in the system output signal is produced. Two spatio-temporal system identification examples are presented to demonstrate the performance of the proposed new modelling framework.

Robust vs. classical principalcomponent analysis in the presence of outliers

Principal Component Analysis (PCA) is a very versatile technique for dimension reduction in multivariate data. Classical PCA is very sensitive to outliers and can lead to misleading conclusions in the presence of outliers. This article studies the merits of robust PCA relative to classical PCA when outliers are present. An algorithm due to Filzmoser et al. (2006) based on a modification of the projection pursuit algorithm of Croux and Ruiz-Gazen (2005) is used for robust PCA computations for a financial data set as well as simulated data sets. Our simulation results indicate that robust PCA generally leads to greater reduction in model dimension than classical PCA in data sets with outliers.

GENERALIZED CELLULAR NEURAL NETWORKS (GCNNs) CONSTRUCTED USING PARTICLE SWARM OPTIMIZATION FOR SPATIO-TEMPORAL EVOLUTIONARY PATTERN IDENTIFICATION

Particle swarm optimization (PSO) is introduced to implement a new constructive learning algorithm for training generalized cellular neural networks (GCNNs) for the identification of spatio-temporal evolutionary (STE) systems. The basic idea of the new PSO-based learning algorithm is to successively approximate the desired signal by progressively pursuing relevant orthogonal projections. This new algorithm will thus be referred to as the orthogonal projection pursuit (OPP) algorithm, which is in mechanism similar to the conventional projection pursuit approach. A novel two-stage hybrid training scheme is proposed for constructing a parsimonious GCNN model. In the first stage, the orthogonal projection pursuit algorithm is applied to adaptively and successively augment the network, where adjustable parameters of the associated units are optimized using a particle swarm optimizer. The resultant network model produced at the first stage may be redundant. In the second stage, a forward orthogonal regression (FOR) algorithm, aided by mutual information estimation, is applied to refine and improve the initially trained network. The effectiveness and performance of the proposed method is validated by applying the new modeling framework to a spatio-temporal evolutionary system identification problem.

A Review of Challenges in the Use of fMRI for Disease Classification / Characterization and A Projection Pursuit Application from Multi-site fMRI Schizophrenia Study.

Functional magnetic resonance imaging (fMRI) is a fairly new technique that has the potential to characterize and classify brain disorders such as schizophrenia. It has the possibility of playing a crucial role in designing objective prognostic/diagnostic tools, but also presents numerous challenges to analysis and interpretation. Classification provides results for individual subjects, rather than results related to group differences. This is a more complicated endeavor that must be approached more carefully and efficient methods should be developed to draw generalized and valid conclusions out of high dimensional data with a limited number of subjects, especially for heterogeneous disorders whose pathophysiology is unknown. Numerous research efforts have been reported in the field using fMRI activation of schizophrenia patients and healthy controls. However, the results are usually not generalizable to larger data sets and require careful definition of the techniques used both in designing algorithms and reporting prediction accuracies. In this review paper, we survey a number of previous reports and also identify possible biases (cross-validation, class size, e.g.) in class comparison/prediction problems. Some suggestions to improve the effectiveness of the presentation of the prediction accuracy results are provided. We also present our own results using a projection pursuit algorithm followed by an application of independent component analysis proposed in an earlier study. We classify schizophrenia versus healthy controls using fMRI data of 155 subjects from two sites obtained during three different tasks. The results are compared in order to investigate the effectiveness of each task and differences between patients with schizophrenia and healthy controls were investigated.

A projection pursuit algorithm for anomaly detection in hyperspectral imagery

The main goal of this paper is to propose an innovative technique for anomaly detection in hyperspectral imageries. This technique allows anomalies to be identified whose signatures are spectrally distinct from their surroundings, without any a priori knowledge of the target spectral signature. It is based on an one-dimensional projection pursuit with the Legendre index as the measure of interest. The index optimization is performed with a simulated annealing over a simplex in order to bypass local optima which could be sub-optimal in certain cases. It is argued that the proposed technique could be considered as seeking a projection to depart from the normal distribution, and unfolding the outliers as a consequence. The algorithm is tested with AHS and HYDICE hyperspectral imageries, where the results show the benefits of the approach in detecting a great variety of objects whose spectral signatures have sufficient deviation from the background. The technique proves to be automatic in the sense that there is no need for parameter tuning, giving meaningful results in all cases. Even objects of sub-pixel size, which cannot be made out by the human naked eye in the original image, can be detected as anomalies. Furthermore, a comparison between the proposed approach and the popular RX technique is given. The former outperforms the latter demonstrating its ability to reduce the proportion of false alarms.

A projection pursuit algorithm to classify individuals using fMRI data: Application to schizophrenia

Schizophrenia is diagnosed based largely upon behavioral symptoms. Currently, no quantitative, biologically based diagnostic technique has yet been developed to identify patients with schizophrenia. Classification of individuals into patient with schizophrenia and healthy control groups based on quantitative biologically based data is of great interest to support and refine psychiatric diagnoses. We applied a novel projection pursuit technique on various components obtained with independent component analysis (ICA) of 70 subjects’ fMRI activation maps obtained during an auditory oddball task. The validity of the technique was tested with a leave-one-out method and the detection performance varied between 80% and 90%. The findings suggest that the proposed data reduction algorithm is effective in classifying individuals into schizophrenia and healthy control groups and may eventually prove useful as a diagnostic tool.

Gaussian mixture model classification: A projection pursuit approach

Gaussian mixture models (GMM) are commonly employed in nonparametric supervised classification. In high-dimensional problems it is often the case that information relevant to the separation of the classes is contained in a few directions. A GMM fitting procedure oriented to supervised classification is proposed, with the aim of reducing the number of free parameters. It resorts to projection pursuit as a dimension reduction method and combines it with GM modelling of class-conditional densities. In its derivation, issues regarding the forward and backward projection pursuit algorithms are discussed. The proposed procedure avoids the “curse of dimensionality”, is able to model structure in subspaces and regularizes the classification model. Its performance is illustrated on a simulation experiment and on a real data set, in comparison with other GMM-based classification methods.

On the forward and backward algorithms of projection pursuit

This article provides a historic review of the forward and backward projection pursuit algorithms, previously thought to be equivalent, and points out an important difference between the two. In doing so, a small error in the original exploratory projection pursuit article by Friedman [J. Amer. Statist. Assoc. 82 1987) 249-266] is corrected. The implication of the difference is briefly discussed in the context of an application in which projection pursuit density estimation is used as a building block for nonparametric discriminant analysis.

Influence of Projection Pursuit on Classification Errors: Computer Simulation Results

Influence of projection pursuit on classification errors and estimates of a posteriori prob- abilities from the sample is considered. Observed random variable is supposed to satisfy a multidi- mensional Gaussian mixture model. Presented computer simulation results show that for compar- atively small sample size classification using projection pursuit algorithm gives better accuracy of estimates of a posteriori probabilities and less classification error.

Multivariate density estimation: A comparative study

This paper is a continuation of the authors' earlier work [1], where a version of the Traven's [2] Gaussian clustering neural network (being a recursive counterpart of the EM algorithm) has been investigated. A comparative simulation study of the Gaussian clustering algorithm [1], two versions of plug-in kernel estimators and a version of Friedman's projection pursuit algorithm are presented for two- and three-dimensional data. Simulations show that the projection pursuit algorithm is a good or a very good estimator, provided, however, that the number of projections is suitably chosen. Although practically confined to estimating normal mixtures, the simulations confirm general reliability of plug-in estimators, and show the same property of the Gaussian clustering algorithm. Indeed, the simulations confirm the earlier conjecture that this last estimator proivdes a way of effectively estimating arbitrary and highly structured continuous densities on Rd, at least for small d, either by using this estimator itself or, rather, by using it as a pilot estimator for a newly proposed plug-in estimator.

A Nonlinear Extension of Principal Component Analysis for Clustering and Spatial Differentiation

The limitations in the use of linear principal component analysis (PCA) for identifying morphological features of data sets are discussed. A nonlinear extension of PCA is introduced to provide this capability. It differs from ordinary PCA methods only in that the objective function involves a nonlinear transformation of the principal components. The procedure is shown to be closely related to the exploratory projection pursuit (EPP) algorithm proposed by Friedman (1987), and thus its usefulness in clustering and spatial differentiation applications is apparent. An efficient gradient ascent algorithm is proposed for implementation, based upon the use of a stochastic approximation. The inherent computational advantages in the suggested implementation over other EPP methods are evident, given that EPP methods require the estimation of a density function at every iteration. The nonlinear extension is evaluated by using several well-known datasets, including Fisher’s (1936) Iris data. An industrial application...

Unsupervised BCM projection pursuit algorithms for classification of simulated radar presentations

A comparison of the unsupervised Projection Pursuit learning algorithm (BCM), with supervised backward propagation (BP) and a laterally inhibited version of BP (LIBP) was performed. Simulated inverse synthetic aperature radar (ISAR) presentations served as a testbed for evaluation. Symmetries of the artificial presentations make the use of localized moments a convenient preprocessing tool for the inputs. Although all three algorithms obtain classification rates comparable to trained human observers for this simulated data base, BCM obtains solutions that classify more effectively inputs that are corrupted by noise or errors in registration; in noise tolerance experiments, the best BCM solution represents a 10 dB improvement over the best BP solution. Recurrent and differential forms of BCM that could be applied to time-dependent classification problems are also developed.

Some Practical Aspects of Exploratory Projection Pursuit

This paper shows how to choose the number of terms in a projection pursuit index and how to select a projection pursuit algorithm. It also demonstrates that Friedman’s index [J. Amer. Statist. Assoc., 82 (1987), pp. 249–266] works better than Hall’s [Ann. Statist. 17 (1989), pp. 589–605] in detecting separations or clusters.

Estimation of a Projection-Pursuit Type Regression Model

Since the pioneering work of Friedman and Stuetzle in 1981, projection-pursuit algorithms have attracted increasing attention. This is mainly due to their potential for overcoming or reducing difficulties arising in nonparametric regression models associated with the so-called curse of dimensionality, that is, the amount of data required to avoid an unacceptably large variance increasing rapidly with dimensionality. Subsequent work has, however, uncovered a dependence on dimensionality for projection-pursuit regression models. Here we propose a projection-pursuit type estimation scheme, with two additional constraints imposed, for which the rate of convergence of the estimator is shown to be independent of the dimensionality. Let $(\mathbf{X}, Y)$ be a random vector such that $\mathbf{X} = (X_1, \ldots, X_d)^T$ ranges over $R^d$. The conditional mean of $Y$ given $\mathbf{X} = \mathbf{x}$ is assumed to be the sum of no more than $d$ general smooth functions of $\beta^T_i\mathbf{x}$, where $\beta_i \in S^{d - 1}$, the unit sphere in $R^d$ centered at the origin. A least-squares polynomial spline and the final prediction error criterion are used to fit the model to a random sample of size $n$ from the distribution of $(\mathbf{X}, Y)$. Under appropriate conditions, the rate of convergence of the proposed estimator is independent of $d$.

Exploratory Projection Pursuit

Abstract A new projection pursuit algorithm for exploring multivariate data is presented that has both statistical and computational advantages over previous methods. A number of practical issues concerning its application are addressed. A connection to multivariate density estimation is established, and its properties are investigated through simulation studies and application to real data. The goal of exploratory projection pursuit is to use the data to find low- (one-, two-, or three-) dimensional projections that provide the most revealing views of the full-dimensional data. With these views the human gift for pattern recognition can be applied to help discover effects that may not have been anticipated in advance. Since linear effects are directly captured by the covariance structure of the variable pairs (which are straightforward to estimate) the emphasis here is on the discovery of nonlinear effects such as clustering or other general nonlinear associations among the variables. Although arbitrary nonlinear effects are impossible to parameterize in full generality, they are easily recognized when presented in a low-dimensional visual representation of the data density. Projection pursuit assigns a numerical index to every projection that is a functional of the projected data density. The intent of this index is to capture the degree of nonlinear structuring present in the projected distribution. The pursuit consists of maximizing this index with respect to the parameters defining the projection. Since it is unlikely that there is only one interesting view of a multivariate data set, this procedure is iterated to find further revealing projections. After each maximizing projection has been found, a transformation is applied to the data that removes the structure present in the solution projection while preserving the multivariate structure that is not captured by it. The projection pursuit algorithm is then applied to these transformed data to find additional views that may yield further insight. This projection pursuit algorithm has potential advantages over other dimensionality reduction methods that are commonly used for data exploration. It focuses directly on the “interestingness” of a projection rather than indirectly through the interpoint distances. This allows it to be unaffected by the scale and (linear) correlational structure of the data, helping it to overcome the “curse of dimensionality” that tends to plague methods based on multidimensional scaling, parametric mapping, cluster analysis, and principal components.