Nonlinear Sufficient Dimension Reduction Research Articles

A selective review of nonlinear sufficient dimension reduction

A selective review of nonlinear sufficient dimension reduction

Nonlinear sufficient dimension reduction for distribution-on-distribution regression

We introduce a new approach to nonlinear sufficient dimension reduction in cases where both the predictor and the response are distributional data, modeled as members of a metric space. Our key step is to build universal kernels (cc-universal) on the metric spaces, which results in reproducing kernel Hilbert spaces for the predictor and response that are rich enough to characterize the conditional independence that determines sufficient dimension reduction. For univariate distributions, we construct the universal kernel using the Wasserstein distance, while for multivariate distributions, we resort to the sliced Wasserstein distance. The sliced Wasserstein distance ensures that the metric space possesses similar topological properties to the Wasserstein space, while also offering significant computation benefits. Numerical results based on synthetic data show that our method outperforms possible competing methods. The method is also applied to several data sets, including fertility and mortality data and Calgary temperature data.

Sliced Inverse Regression in Metric Spaces

In this article, we propose a general nonlinear sufficient dimension reduction (SDR) framework when both the predictor and response lie in some general metric spaces. We construct reproducing kernel Hilbert spaces whose kernels are fully determined by the distance functions of the metric spaces, then leverage the inherent structures of these spaces to define a nonlinear SDR framework. We adapt the classical sliced inverse regression of \citet{Li:1991} within this framework for the metric space data. We build the estimator based on the corresponding linear operators, and show it recovers the regression information unbiasedly. We derive the estimator at both the operator level and under a coordinate system, and also establish its convergence rate. We illustrate the proposed method with both synthetic and real datasets exhibiting non-Euclidean geometry.

A new reproducing kernel‐based nonlinear dimension reduction method for survival data

AbstractBased on the theories of sliced inverse regression (SIR) and reproducing kernel Hilbert space (RKHS), a new approach RDSIR (RKHS‐based Double SIR) to nonlinear dimension reduction for survival data is proposed. An isometric isomorphism is constructed based on the RKHS property, then the nonlinear function in the RKHS can be represented by the inner product of two elements that reside in the isomorphic feature space. Due to the censorship of survival data, double slicing is used to estimate the weight function to adjust for the censoring bias. The nonlinear sufficient dimension reduction (SDR) subspace is estimated by a generalized eigen‐decomposition problem. The asymptotic property of the estimator is established based on the perturbation theory. Finally, the performance of RDSIR is illustrated on simulated and real data. The numerical results show that RDSIR is comparable with the linear SDR method. Most importantly, RDSIR can also effectively extract nonlinearity from survival data.

On a nonlinear extension of the principal fitted component model

We propose a nonlinear sufficient dimension reduction method called the kernel principal fitted component model using the kernel method under a reproducing kernel Hilbert space. The kernel principal fitted component is a nonlinear extension of the principal fitted component model, and it is found in the theory of mapping low dimensional input space to the higher dimensional feature space so that we can apply well-developed linear methods to the nonlinear dataset. We derive our method coincides with the generalized sliced inverse regression under some mild assumptions and show the dimension reduction subspace extracted from the kernel principal fitted component model is contained in the central class. In the numerical experiments, we present the kernel principal fitted component model with the Gaussian kernel can extract the linear and nonlinear features well for the models from both forward and inverse regression settings. By applying our method to ovarian cancer microarray dataset, we demonstrate the kernel principal fitted component can provide a competitive prediction accuracy and computational efficiency in the high-dimensional classification problem.

On sufficient dimension reduction via principal asymmetric least squares

Principal asymmetric least squares (PALS) is introduced as a novel method for sufficient dimension reduction with heteroscedastic error. Classical methods such as MAVE [Xia et al. (2002), ‘An Adaptive Estimation of Dimension Reduction Space’ (with discussion), Journal of the Royal Statistical Society Series B, 64, 363–410] and PSVM [Li et al. (2011), ‘Principal Support Vector Machines for Linear and Nonlinear Sufficient Dimension Reduction’, The Annals of Statistics, 39, 3182–3210] may not perform well in the presence of heteroscedasticity, while the new proposal addresses this limitation by synthesising different expectile levels. Through extensive numerical studies, we demonstrate the superior performance of PALS in terms of estimation accuracy over classical methods including MAVE and PSVM. For the asymptotic analysis of PALS, we develop new tools to compute the derivative of an expectation of a non-Lipschitz function.

Nonlinear sufficient dimension reduction for functional data

We propose a general theory and the estimation procedures for nonlinear sufficient dimension reduction where both the predictor and the response may be random functions. The relation between the response and predictor can be arbitrary and the sets of observed time points can vary from subject to subject. The functional and nonlinear nature of the problem leads to construction of two functional spaces: the first representing the functional data, assumed to be a Hilbert space, and the second characterizing nonlinearity, assumed to be a reproducing kernel Hilbert space. A particularly attractive feature of our construction is that the two spaces are nested, in the sense that the kernel for the second space is determined by the inner product of the first. We propose two estimators for this general dimension reduction problem, and establish the consistency and convergence rate for one of them. These asymptotic results are flexible enough to accommodate both fully and partially observed functional data. We investigate the performances of our estimators by simulations, and applied them to data sets about speech recognition and handwritten symbols.

Statistical learning on emerging economies

ABSTRACTBRIC is an acronym coined by Jim O'Neill from Goldman Sachs in 2001 to abbreviate four emerging economies, Brazil, Russia, India and China, based on economic data at the time. Later, as new data became available, Goldman Sachs updated this list to include Mexico, Indonesia, Nigeria and Turkey, which was referred to as MINT. This list, as well as some other similar lists of emerging economies, is based on descriptive statistics of the economic data combined with economists' insights. The purpose of this study is twofold: to see if these insights into the global economic trends can be learned with statistical learning tools, and, if so, to identify the next emerging countries. We apply both unsupervised and supervised learning methods, which include linear and nonlinear principle component analysis, and nonlinear sufficient dimension reduction, to 13 years worth of economic data. Our results show that these statistical learning techniques, and in particular the kernel sliced inverse regression algorithm, can serve as a useful tool for economists and policy-makers for analyzing global economic trends, by its ability to incorporate large amount of economic data and previous experts' judgments, which otherwise may take years of experiences to acquire.

Principal weighted support vector machines for sufficient dimension reduction in binary classification.

SUMMARYSufficient dimension reduction is popular for reducing data dimensionality without stringent model assumptions. However, most existing methods may work poorly for binary classification. For example, sliced inverse regression (Li, 1991) can estimate at most one direction if the response is binary. In this paper we propose principal weighted support vector machines, a unified framework for linear and nonlinear sufficient dimension reduction in binary classification. Its asymptotic properties are studied, and an efficient computing algorithm is proposed. Numerical examples demonstrate its performance in binary classification.

Sufficient dimension reduction via principal L$q$ support vector machine

Principal support vector machine was proposed recently by Li, Artemiou and Li (2011) to combine L$1$ support vector machine and sufficient dimension reduction. We introduce the principal L$q$ support vector machine as a unified framework for linear and nonlinear sufficient dimension reduction. By noticing that the solution of L$1$ support vector machine may not be unique, we set $q>1$ to ensure the uniqueness of the solution. The asymptotic distribution of the proposed estimators are derived for $q> 1$. We demonstrate through numerical studies that the proposed L$2$ support vector machine estimators improve existing methods in accuracy, and are less sensitive to the tuning parameter selection.

A general theory for nonlinear sufficient dimension reduction: Formulation and estimation

In this paper we introduce a general theory for nonlinear sufficient dimension reduction, and explore its ramifications and scope. This theory subsumes recent work employing reproducing kernel Hilbert spaces, and reveals many parallels between linear and nonlinear sufficient dimension reduction. Using these parallels we analyze the properties of existing methods and develop new ones. We begin by characterizing dimension reduction at the general level of $\sigma$-fields and proceed to that of classes of functions, leading to the notions of sufficient, complete and central dimension reduction classes. We show that, when it exists, the complete and sufficient class coincides with the central class, and can be unbiasedly and exhaustively estimated by a generalized sliced inverse regression estimator (GSIR). When completeness does not hold, this estimator captures only part of the central class. However, in these cases we show that a generalized sliced average variance estimator (GSAVE) can capture a larger portion of the class. Both estimators require no numerical optimization because they can be computed by spectral decomposition of linear operators. Finally, we compare our estimators with existing methods by simulation and on actual data sets.

Principal support vector machines for linear and nonlinear sufficient dimension reduction

We introduce a principal support vector machine (PSVM) approach that can be used for both linear and nonlinear sufficient dimension reduction. The basic idea is to divide the response variables into slices and use a modified form of support vector machine to find the optimal hyperplanes that separate them. These optimal hyperplanes are then aligned by the principal components of their normal vectors. It is proved that the aligned normal vectors provide an unbiased, $\sqrt{n}$-consistent, and asymptotically normal estimator of the sufficient dimension reduction space. The method is then generalized to nonlinear sufficient dimension reduction using the reproducing kernel Hilbert space. In that context, the aligned normal vectors become functions and it is proved that they are unbiased in the sense that they are functions of the true nonlinear sufficient predictors. We compare PSVM with other sufficient dimension reduction methods by simulation and in real data analysis, and through both comparisons firmly establish its practical advantages.