Projection Pursuit Learning Network Research Articles

A new approach for unsupervised restoring images based on Wavelet-Domain Projection Pursuit Learning Network

The Wavelet-Domain Projection Pursuit Learning Network (WDPPLN) is proposed for restoring degraded image. The new network combines the advantages of both projection pursuit and wavelet shrinkage. Restoring image is very difficult when little is known about a priori knowledge for multisource degraded factors. WDPPLN successfully resolves this problem by separately processing wavelet coefficients and scale coefficients. Parameters in WDPPLN, which are used to simulate degraded factors, are estimated via WDPPLN training, using scale coefficients. Also, WDPPLN uses soft-threshold of wavelet shrinkage technique to suppress noise in three high frequency subbands. The new method is compared with the traditional methods and the Projection Pursuit Learning Network (PPLN) method. Experimental results demonstrate that it is an effective method for unsupervised restoring degraded image.

Sigma-Pi Cascade Extended Hybrid Neural Network

A nonparametric neural architecture called the Sigma-Pi Cascade extended Hybrid Neural Network σπ-(CHNN) is proposed to extend approximation capabilities in neural architectures such as Projection Pursuit Learning (PPL) and Hybrid Neural Networks (HNN). Like PPL and HNN, σπ-CHNN also uses distinct activation functions in its neurons but, unlike these previous neural architectures, it may consider multiplicative operators in its hidden neurons, enabling it to extract higher-order information from given data. σπ-CHNN uses arbitrary connectivity patterns among neurons. An evolutionary learning algorithm combined with a conjugate gradient algorithm is proposed to automatically design the topology and weights of σπ-CHNN. σπ-CHNN performance is evaluated in five benchmark regression problems. Results show that σπ-CHNN provides competitive performance compared to PPL and HNN in most problems, either in computational requirements to implement the proposed neural architecture or in approximation accuracy. In some problems, σπ-CHNN reduces the approximation error on the order of 10-1 compared to PPL and HNN, whereas in other cases it achieves the same approximation error as these neural architectures but uses a smaller number of hidden neurons (usually 1 hidden neuron less than PPL and HNN).

A hybrid learning system for image deblurring

In this paper we propose a 3-stage hybrid learning system with unsupervised learning to cluster data in the first stage, supervised learning in the middle stage to determine network parameters and finally a decision-making stage using voting mechanism. We take this opportunity to study the role of various supervised learning systems that constitute the middle stage. Specifically, we focus on one-hidden layer neural network with sigmoidal activation function, radial basis function network with Gaussian activation function and projection pursuit learning network with Hermite polynomial as the activation function. These learning systems rank in increasing order of complexity. We train and test each system with identical data sets. Experimental results show that learning ability of a system is controlled by the shape of the activation function when other parameters remain fixed. We observe that clustering in the input space leads to better system performance. Experimental results provide compelling evidences in favor of use of the hybrid learning system and the committee machines with gating network.

Application of projection pursuit learning to boundary detection and deblurring in images

Projection pursuit learning networks (PPLNs) have been used in many fields of research but have not been widely used in image processing. In this paper we demonstrate how this highly promising technique may be used to connect edges and produce continuous boundaries. We also propose the application of PPLN to deblurring a degraded image when little or no a priori information about the blur is available. The PPLN was successful at developing an inverse blur filter to enhance blurry images. Theory and background information on projection pursuit regression (PPR) and PPLN are also presented.

Application of projection pursuit learning networks in dynamic system modeling

A new type of feedforward neural network, PPLN (Projection Pursuit Learning NetWork), is introduced into the context of control system modeling. The mathematical principle and the learning strategy of PPLN are presented, and its characteristics is analyzed. PPLN is suitable for solving high dimension problems in that it is parsimonious and adaptive to small training data set. Simulation with CSTR show that the dynamic system is successfully identified. PPLN is proved have exciting advantages for modeling of mulli-variable control systems

Use of bias term in projection pursuit learning improves approximation and convergence properties

In a regression problem, one is given a multidimensional random vector X, the components of which are called predictor variables, and a random variable, Y, called response. A regression surface describes a general relationship between X and Y. A nonparametric regression technique that has been successfully applied to high-dimensional data is projection pursuit regression (PPR). The regression surface is approximated by a sum of empirically determined univariate functions of linear combinations of the predictors. Projection pursuit learning (PPL) formulates PPR using a 2-layer feedforward neural network. The smoothers in PPR are nonparametric, whereas those in PPL are based on Hermite functions of some predefined highest order R. We demonstrate that PPL networks in the original form do not have the universal approximation property for any finite R, and thus cannot converge to the desired function even with an arbitrarily large number of hidden units. But, by including a bias term in each linear projection of the predictor variables, PPL networks can regain these capabilities, independent of the exact choice of R. Experimentally, it is shown in this paper that this modification increases the rate of convergence with respect to the number of hidden units, improves the generalization performance, and makes it less sensitive to the setting of R. Finally, we apply PPL to chaotic time series prediction, and obtain superior results compared with the cascade-correlation architecture.

The cascade-correlation learning: a projection pursuit learning perspective

Cascade-correlation (Cascor) is a popular supervised learning architecture that dynamically grows layers of hidden neurons of fixed nonlinear activations (e.g., sigmoids), so that the network topology (size, depth) can be efficiently determined. Similar to a cascade-correlation learning network (CCLN), a projection pursuit learning network (PPLN) also dynamically grows the hidden neurons. Unlike a CCLN where cascaded connections from the existing hidden units to the new candidate hidden unit are required to establish high-order nonlinearity in approximating the residual error, a PPLN approximates the high-order nonlinearity by using trainable parametric or semi-parametric nonlinear smooth activations based on minimum mean squared error criterion. An analysis is provided to show that the maximum correlation training criterion used in a CCLN tends to produce hidden units that saturate and thus makes it more suitable for classification tasks instead of regression tasks as evidenced in the simulation results. It is also observed that this critical weakness in CCLN can also potentially carry over to classification tasks, such as the two-spiral benchmark used in the original CCLN paper.

Implementing projection pursuit learning

This paper examines the implementation of projection pursuit regression (PPR) in the context of machine learning and neural networks. We propose a parametric PPR with direct training which achieves improved training speed and accuracy when compared with nonparametric PPR. Analysis and simulations are done for heuristics to choose good initial projection directions. A comparison of a projection pursuit learning network with a single hidden-layer sigmoidal neural network shows why grouping hidden units in a projection pursuit learning network is useful. Learning robot arm inverse dynamics is used as an example problem.

Improving the approximation and convergence capabilities of projection pursuit learning

One nonparametric regression technique that has been successfully applied to high-dimensional data is projection pursuit regression (PPR). In this method, the regression surface is approximated by a sum of empirically determined univariate functions of linear combinations of the predictors. Projection pursuit learning (PPL) proposed by Hwanget al. formulates PPR using a two-layer feedforward neural network. One of the main differences between PPR and PPL is that the smoothers in PPR are nonparametric, whereas those in PPL are based on Hermite functions of some predefined highest orderR. While the convergence property of PPR is already known, that for PPL has not been thoroughly studied. In this paper, we demonstrate that PPL networks do not have the universal approximation and strong convergence properties for any finiteR. But, by including a bias term in each linear combination of the predictor variables, PPL networks can regain these capabilities, independent of the exact choice ofR. It is also shown experimentally that this modification improves the generalization performance in regression problems, and creates smoother decision surfaces for classification problems.

Projection pursuit learning networks for regression

Two types of feedforward network learning methods for model-free regression problems are studied and compared in this paper: one is the popular back-propagation learning (BPL) which is well known in the artificial neural network literature; the other is the projection pursuit learning (PPL) which has emerged in recent years in the statistical estimation literature. Both the BPL and the PPL are based on projections of the data in directions determined from interconnection weights. However, unlike the totally parametric BPL method, which employs a fixed set of (usually sigmoidal) nonlinear nodal functions in obtaining an estimate of all the weights simultaneously at each iteration, the PPL nonparametrically estimates unknown nonlinear functions sequentially (neuron-by-neuron and layer-by-layer) at each iteration while jointly estimating the interconnection weights. In terms of learning efficiency, both methods have comparable training speeds when based on a Gauss-Newton optimization algorithm, while the PPL is more parsimonious, i.e. in order to achieve comparable accuracy, the PPL requires fewer hidden neurons to approximate the true function. In terms of learning robustness toward noise outliers, the BPL is more sensitive to the outliers.