Dimension Reduction Subspace Research Articles

Gradient‐based approach to sufficient dimension reduction with functional or longitudinal covariates

AbstractIn this paper, we focus on the sufficient dimension reduction problem in regression analysis with real‐valued response and functional or longitudinal covariates. We propose a new method based on gradients of the conditional distribution function to estimate the sufficient dimension reduction subspace. While existing inverse‐regression‐type methods relies on a linearity condition, our method is based on the gradient of conditional distribution function and its validity only requires smoothness conditions on the population parameters. Practically, the proposed estimator can be obtained by standard algorithm of functional principal component analysis. The proposed method is demonstrated through extensive simulations and two empirical examples.

A new reproducing kernel‐based nonlinear dimension reduction method for survival data

AbstractBased on the theories of sliced inverse regression (SIR) and reproducing kernel Hilbert space (RKHS), a new approach RDSIR (RKHS‐based Double SIR) to nonlinear dimension reduction for survival data is proposed. An isometric isomorphism is constructed based on the RKHS property, then the nonlinear function in the RKHS can be represented by the inner product of two elements that reside in the isomorphic feature space. Due to the censorship of survival data, double slicing is used to estimate the weight function to adjust for the censoring bias. The nonlinear sufficient dimension reduction (SDR) subspace is estimated by a generalized eigen‐decomposition problem. The asymptotic property of the estimator is established based on the perturbation theory. Finally, the performance of RDSIR is illustrated on simulated and real data. The numerical results show that RDSIR is comparable with the linear SDR method. Most importantly, RDSIR can also effectively extract nonlinearity from survival data.

On a nonlinear extension of the principal fitted component model

We propose a nonlinear sufficient dimension reduction method called the kernel principal fitted component model using the kernel method under a reproducing kernel Hilbert space. The kernel principal fitted component is a nonlinear extension of the principal fitted component model, and it is found in the theory of mapping low dimensional input space to the higher dimensional feature space so that we can apply well-developed linear methods to the nonlinear dataset. We derive our method coincides with the generalized sliced inverse regression under some mild assumptions and show the dimension reduction subspace extracted from the kernel principal fitted component model is contained in the central class. In the numerical experiments, we present the kernel principal fitted component model with the Gaussian kernel can extract the linear and nonlinear features well for the models from both forward and inverse regression settings. By applying our method to ovarian cancer microarray dataset, we demonstrate the kernel principal fitted component can provide a competitive prediction accuracy and computational efficiency in the high-dimensional classification problem.

Inference for the dimension of a regression relationship using pseudo-covariates.

In data analysis using dimension reduction methods, the main goal is to summarize how the response is related to the covariates through a few linear combinations. One key issue is to determine the number of independent, relevant covariate combinations, which is the dimension of the sufficient dimension reduction (SDR) subspace. In this work, we propose an easily-applied approach to conduct inference for the dimension of the SDR subspace, based on augmentation of the covariate set with simulated pseudo-covariates. Applying the partitioning principal to the possible dimensions, we use rigorous sequential testing to select the dimensionality, by comparing the strength of the signal arising from the actual covariates to that appearing to arise from the pseudo-covariates. We show that under a "uniform direction" condition, our approach can be used in conjunction with several popular SDR methods, including sliced inverse regression. In these settings, the test statistic asymptotically follows a beta distribution and therefore is easily calibrated. Moreover, the family-wise type I error rate of our sequential testing is rigorously controlled. Simulation studies and an analysis of newborn anthropometric data demonstrate the robustness of the proposed approach, and indicate that the power is comparable to or greater than thealternatives.

Strong consistency of kernel method for sliced average variance estimation

In this paper, we consider the kernel method to estimate sliced average variance estimation (SAVE). SAVE is a tool as SIR (sliced inverse regression), recommended to identify and to estimate the central dimension reduction (CDR) subspace. CDR subspace is the intersection of all dimension reduction subspaces which are at the base to describe the conditional distribution of the response Y given a dimensional predictor vector X. SAVE and even SIR are used to estimate CDR subspace. Two versions are very popular: slice version and kernel version. In this paper, we are looking at the kernel version. For Kernel SAVE version, two asymptotic properties have been demonstrated in particular: asymptotic normality and convergence in probability. And we know that these two properties, although important, are weak in front of almost sure convergence. However, until now, the strong consistency has not yet been obtained. In this paper, we obtain, under weaker assumptions, this asymptotic property.

On Efficient Dimension Reduction with Respect to the Interaction between Two Response Variables

AbstractIn this paper, we propose the novel theory and methodologies for dimension reduction with respect to the interaction between two response variables, which is a new research problem that has wide applications in missing data analysis, causal inference, graphical models, etc. We formulate the parameters of interest to be the locally and the globally efficient dimension reduction subspaces, and justify the generality of the corresponding low-dimensional assumption. We then construct estimating equations that characterize these parameters, using which we develop a generic family of consistent, model-free and easily implementable dimension reduction methods called the dual inverse regression methods. We also build the theory regarding the existence of the globally efficient dimension reduction subspace, and provide a handy way to check this in practice. The proposed work differs fundamentally from the literature of sufficient dimension reduction in terms of the research interest, the assumption adopted, the estimation methods and the corresponding applications, and it potentially creates a new paradigm of dimension reduction research. Its usefulness is illustrated by simulation studies and a real data example at the end.

Direct Position Determination of Noncircular Sources with Multiple Nested Arrays: Reduced Dimension Subspace Data Fusion

Direct position determination (DPD) of noncircular (NC) sources with multiple nested arrays (NA) is investigated in this paper. Noncircular sources are used to expand the dimension of the received signal matrix, so the number of identifiable information sources and the accuracy of direct position determination are improved. Furthermore, nested array increases spatial degree of freedom. In this paper, the high‐dimensional search problem of noncircular sources is investigated. Therefore, we propose algorithm dimension reduction subspace data fusion (RD‐SDF) to reduce complexity and increase positioning accuracy. Simulation results show that the proposed RD‐SDF algorithm for multiple nested arrays with noncircular sources has improved positioning accuracy with higher spatial degree of freedom than SDF, Capon, and two‐step algorithms with uniform linear array and circular sources (CS).

Structural reliability assessment through surrogate based importance sampling with dimension reduction

We present a method for reliability assessment in extreme conditions from a numerical simulator through surrogate based importance sampling. As proposed in recent works in the literature, a Kriging surrogate is used to build an approximation of the limit state function and the optimal importance density. Our contribution is then the use of a sufficient dimension reduction method which enables the construction of the limit state function metamodel in lower dimension. The so called augmented failure probability and correction factor are recast in this dimension reduction framework. Simple strategies for metamodel refinement in the dimension reduction subspace are described and, in the case of Gaussian inputs, a computationally efficient MCMC scheme aimed at sampling the quasi-optimal importance density is presented. The case of non-Gaussian inputs is also laid out and it is argued and demonstrated through simulations that this approach can reduce the number of calls to the computer model, which is a crucial factor in reliability analysis. Advantages of this method are also supported by numerical simulations carried on an industrial case study concerned with the extreme response prediction of a wind turbine under wind loading.

Partial Dynamic Dimension Reduction for Conditional Mean in Regression

In many regression analysis, the authors are interested in regression mean of response variate given predictors, not its the conditional distribution. This paper is concerned with dimension reduction of predictors in sense of mean function of response conditioning on predictors. The authors introduce the notion of partial dynamic central mean dimension reduction subspace, different from central mean dimension reduction subspace, it has varying subspace in the domain of predictors, and its structural dimensionality may not be the same point by point. The authors study the property of partial dynamic central mean dimension reduction subspace, and develop estimated methods called dynamic ordinary least squares and dynamic principal Hessian directions, which are extension of ordinary least squares and principal Hessian directions based on central mean dimension reduction subspace. The kernel estimate methods for dynamic ordinary least squares and dynamic Principal Hessian Directions are employed, and large sample properties of estimators are given under the regular conditions. Simulations and real data analysis demonstrate that they are effective.

Double-slicing assisted sufficient dimension reduction for high-dimensional censored data

This paper provides a unified framework and an efficient algorithm for analyzing high-dimensional survival data under weak modeling assumptions. In particular, it imposes neither parametric distributional assumption nor linear regression assumption. It only assumes that the survival time $T$ depends on a high-dimensional covariate vector $\mathbf{X}$ through low-dimensional linear combinations of covariates $\Gamma ^{T}\mathbf{X}$. The censoring time is allowed to be conditionally independent of the survival time given the covariates. This general framework includes many popular parametric and semiparametric survival regression models as special cases. The proposed algorithm produces a number of practically useful outputs with theoretical guarantees, including a consistent estimate of the sufficient dimension reduction subspace of $T\mid \mathbf{X}$, a uniformly consistent Kaplan–Meier-type estimator of the conditional distribution function of $T$ and a consistent estimator of the conditional quantile survival time. Our asymptotic results significantly extend the classical theory of sufficient dimension reduction for censored data (particularly that of Li, Wang and Chen in Ann. Statist. 27 (1999) 1–23) and the celebrated nonparametric Kaplan–Meier estimator to the setting where the number of covariates $p$ diverges exponentially fast with the sample size $n$. We demonstrate the promising performance of the proposed new estimators through simulations and a real data example.

An enhanced Kriging surrogate modeling technique for high-dimensional problems

Surrogate modeling techniques are widely used to simulate the behavior of manufactured and engineering systems. The construction of such surrogate models may become intractable in cases when input spaces have high dimensions, because the large number of model responses is typically required to estimate model parameters. In this paper, we proposed a new Kriging modeling technique combined with dimension reduction method to address the issue. In the proposed method, the sliced inverse regression technique is utilized to achieve a dimension reduction by constructing a new projection vector which reduces the dimension of the original input vector without losing the essential information of the model response quantify of interest. In the dimension reduction subspace, a new correlation function of Kriging is constructed by means of the tensor product of several correlation functions with respect to each projection direction. The proposed method is especially promising for high-dimensional problems. In examples including finite element model (FEM) pertinent to low cycle fatigue life (LCF) of a aero-engine compressor disc, the enhanced Kriging is found to outperform several well-established surrogate models when small sample sizes are used.

Dimension reduction for functional regression with a binary response

We propose here a novel functional inverse regression method (i.e., functional surrogate assisted slicing) for functional data with binary responses. Previously developed method (e.g., functional sliced inverse regression) can detect no more than one direction in the functional sufficient dimension reduction subspace. In contrast, the proposed new method can detect multiple directions. The population properties of the proposed method is established. Furthermore, we propose a new method to estimate the functional central space which do not need the inverse of the covariance operator. To practically determine the structure dimension of the functional sufficient dimension reduction subspace, a modified Bayesian information criterion method is proposed. Numerical studies based on both simulated and real data sets are presented.

Counting process-based dimension reduction methods for censored outcomes.

We propose counting process-based dimension reduction methods for right-censored survival data. Semiparametric estimating equations are constructed to estimate the dimension reduction subspace for the failure time model. Our methods address two limitations of existing approaches. First, using the counting process formulation, they do not require estimation of the censoring distribution to compensate for the bias in estimating the dimension reduction subspace. Second, the nonparametric estimation involved adapts to the structural dimension, so our methods circumvent the curse of dimensionality. Asymptotic normality is established for the estimators. We propose a computationally efficient approach that requires only a singular value decomposition to estimate the dimension reduction subspace. Numerical studies suggest that our new approaches exhibit significantly improved performance. The methods are implemented in the [Formula: see text] package [Formula: see text].

SAVE: Robust or not?

Sliced average variance estimation (SAVE) is one of the best methods for estimating central dimension-reduction subspace in semi parametric regression models when covariates are normal. In recent days SAVE is being used to analyze DNA microarray data especially in tumor classification but most important drawback is normality of covariates. In this article, the asymptotic behavior of estimates of CDR space under varying slice size is studied through simulation studies when covariates are non normal but follows linearity condition as well as when covariates slightly perturbed from normal distribution and we observed that serious error may occur under violation normality assumption.

Efficient Integration of Sufficient Dimension Reduction and Prediction in Discriminant Analysis

ABSTRACTSufficient dimension reduction (SDR) methods are popular model-free tools for preprocessing and data visualization in regression problems where the number of variables is large. Unfortunately, reduce-and-classify approaches in discriminant analysis usually cannot guarantee improvement in classification accuracy, mainly due to the different nature of the two stages. On the other hand, envelope methods construct targeted dimension reduction subspaces that achieve dimension reduction and improve parameter estimation efficiency at the same time. However, little is known about how to construct envelopes in discriminant analysis models. In this article, we introduce the notion of the envelope discriminant subspace (ENDS) as a natural inferential and estimative object in discriminant analysis that incorporates these considerations. We develop the ENDS estimators that simultaneously achieve sufficient dimension reduction and classification. Consistency and asymptotic normality of the ENDS estimators are established, where we carefully examine the asymptotic efficiency gain under the classical linear and quadratic discriminant analysis models. Simulations and real data examples show superb performance of the proposed method. Supplementary materials for this article are available online.

Robust dimension reduction using sliced inverse median regression

Dimension reduction is a useful technique when working with high-dimensional predictors, as meaningful data visualizations and graphical analyses using fewer predictors can be achieved. We propose a new non-iterative and robust against extreme values estimation of the effective dimension reduction (e.d.r) subspace, which is based on the estimation of the conditional median function of the predictors given the response. The existing literature on robust estimation of the e.d.r subspace relies on iterative algorithms, such as the composite quantile minimum average variance estimation and the sliced regression. Compared with these existing robust dimension reduction methods, the new method avoids iterations by directly estimating the e.d.r subspace and has better finite sample performance. It is shown that the inverse Tukey and Oja median regression curve falls into the e.d.r subspace, and that its directions can be estimated $$\sqrt{n}$$ -consistently.

Model-free envelope dimension selection

An envelope is a targeted dimension reduction subspace for simultaneously achieving dimension reduction and improving parameter estimation efficiency. While many envelope methods have been proposed in recent years, all envelope methods hinge on the knowledge of a key hyperparameter, the structural dimension of the envelope. How to estimate the envelope dimension consistently is of substantial interest from both theoretical and practical aspects. Moreover, very recent advances in the literature have generalized envelope as a model-free method, which makes selecting the envelope dimension even more challenging. Likelihood-based approaches such as information criteria and likelihood-ratio tests either cannot be directly applied or have no theoretical justification. To address this critical issue of dimension selection, we propose two unified approaches – called FG and 1D selections – for determining the envelope dimension that can be applied to any envelope models and methods. The two model-free selection approaches are based on the two different envelope optimization procedures: the full Grassmannian (FG) optimization and the 1D algorithm [11], and are shown to be capable of correctly identifying the structural dimension with a probability tending to 1 under mild moment conditions as the sample size increases. While the FG selection unifies and generalizes the BIC and modified BIC approaches that existing in the literature, and hence provides the theoretical justification of them under weak moment condition and model-free context, the 1D selection is computationally more stable and efficient in finite sample. Extensive simulations and a real data analysis demonstrate the superb performance of our proposals.

Response dimension reduction: model-based approach

ABSTRACTIn this paper, a model-based approach to reduce the dimension of response variables in multivariate regression is newly proposed, following the existing context of the response dimension reduction developed by Yoo and Cook [Response dimension reduction for the conditional mean in multivariate regression. Comput Statist Data Anal. 2008;53:334–343]. The related dimension reduction subspace is estimated by maximum likelihood, assuming an additive error. In the new approach, the linearity condition, which is assumed for the methodological development in Yoo and Cook (2008), is understood through the covariance matrix of the random error. Numerical studies show potential advantages of the proposed approach over Yoo and Cook (2008). A real data example is presented for illustration.

Minimum average deviance estimation for sufficient dimension reduction

ABSTRACTSufficient dimension reduction methods aim to reduce the dimensionality of predictors while preserving regression information relevant to the response. In this article, we develop Minimum Average Deviance Estimation (MADE) methodology for sufficient dimension reduction. The purpose of MADE is to generalize Minimum Average Variance Estimation (MAVE) beyond its assumption of additive errors to settings where the outcome follows an exponential family distribution. As in MAVE, a local likelihood approach is used to learn the form of the regression function from the data and the main parameter of interest is a dimension reduction subspace. To estimate this parameter within its natural space, we propose an iterative algorithm where one step utilizes optimization on the Stiefel manifold. MAVE is seen to be a special case of MADE in the case of Gaussian outcomes with a common variance. Several procedures are considered to estimate the reduced dimension and to predict the outcome for an arbitrary covariate value. Initial simulations and data analysis examples yield encouraging results and invite further exploration of the methodology.

Functional envelope for model-free sufficient dimension reduction

In this article, we introduce the functional envelope for sufficient dimension reduction and regression with functional and longitudinal data. Functional sufficient dimension reduction methods, especially the inverse regression estimation family of methods, usually involve solving generalized eigenvalue problems and inverting the infinite-dimensional covariance operator. With the notion of functional envelope, essentially a special type of sufficient dimension reduction subspace, we develop a generic method to circumvent the difficulties in solving the generalized eigenvalue problems and inverting the covariance directly. We derive the geometric characteristics of the functional envelope and establish the asymptotic properties of related functional envelope estimators under mild conditions. The functional envelope estimators have shown promising performance in extensive simulation studies and real data analysis.