Unification of neural and statistical modeling methods that combine inputs by linear projection

Abstract

Empirical modeling methods that combine inputs by linear projection include linear methods such as, ordinary least-squares regression, partial least-squares regression, principal components regression, and nonlinear methods such as, backpropagation networks with a single hidden layer, projection pursuit regression, nonlinear partial least-squares regression, and nonlinear principal components regression. In this paper, these popular modeling techniques are unified to yield a single method called nonlinear continuum regression (NLCR). This unification is based on the insight provided by a common framework for empirical modeling methods, and is achieved by using activation functions that adapt to the measured data, a common optimization criterion for finding the projection directions, and a hierarchical training methodology that allows efficient modeling. The adaptive-shape activation functions are determined by univariate smoothing in the space of the projected input versus output. The NLCR optimization criterion contains an adjustable parameter that controls the degree of overfitting or bias of the model, and spans the continuum of methods from projection pursuit regression or backpropagation networks to nonlinear principal components regression. Consequently, NLCR results in models that are usually more general and compact than those obtained by existing methods based on linear projection, while eliminating the need for arbitrary selection of an empirical modeling method based on linear projection for a given task. The improved modeling ability of NLCR and its performance on different types of training data are illustrated by examples based on simulated and industrial data.