Sufficient Dimension Reduction Methods Research Articles

Using noise filtering and sufficient dimension reduction method on unstructured economic data

Using noise filtering and sufficient dimension reduction method on unstructured economic data

Quantitative trading models based on Sufficient Dimension Reduction and Ensemble Learning

With the continuous development of the securities market, constructing quantitative trading strategies with strong generalization ability the adaption to dynamic and changeable market environment is becoming more and more momentous in the field of quantitative finance. Under such circumstances, this paper proposes a stock prediction model based on sufficient dimension reduction theory and ensemble learning, which can be deployed on quantitative trading strategy. The proposed model on the one hand alleviates the curse of dimension by using the sufficient dimension reduction method and maximally retains variation of stock factors, on the other hand employs ensemble learning technique which can weigh model bias and variance to address the overfitting or underfitting problems. Furthermore, since ensemble learning are model free, different sub-models can be applied to improve the generalization ability of the algorithm on predicting occasions. In the section of accuracy comparison and quantitative trading strategy experiments based on real stock datasets, the proposed model demonstrates the best prediction results and adequate robustness compared to other existed methods.

Semiparametric recovery of central dimension reduction space with nonignorable nonresponse

Sufficient dimension reduction (SDR) methods are effective tools for handling high dimensional data. Classical SDR methods are developed under the assumption that the data are completely observed. When the data are incomplete due to missing values, SDR has only been considered when the data are randomly missing, but not when they are nonignorably missing, which is arguably more difficult to handle due to the missing values' dependence on the reasons they are missing. The purpose of this paper is to fill this void. We propose an intuitive, easy‐to‐implement SDR estimator based on a semiparametric propensity score function for response data with non‐ignorable missing values. We refer to it as the dimension reduction‐based imputed estimator. We establish the theoretical properties of this estimator and examine its empirical performance via an extensive numerical study on real and simulated data. As well, we compare the performance of our proposed dimension reduction‐based imputed estimator with two competing estimators, including the fusion refined estimator and cumulative slicing estimator. A distinguishing feature of our method is that it requires no validation sample. The SDR theory developed in this paper is a non‐trivial extension of the existing literature, due to the technical challenges posed by nonignorable missingness. All the technical proofs of the theorems are given in the Appendix .

Aggregate Kernel Inverse Regression Estimation

Sufficient dimension reduction (SDR) is a useful tool for nonparametric regression with high-dimensional predictors. Many existing SDR methods rely on some assumptions about the distribution of predictors. Wang et al. proposed an aggregate dimension reduction method to reduce the dependence on the distributional assumptions. Motivated by their work, we propose a novel and effective method by combining the aggregate method and the kernel inverse regression estimation. The proposed approach can accurately estimate the dimension reduction directions and substantially improve the exhaustivity of the estimates with complex models. At the same time, this method does not depend on the arrangement of slices, and the influence of the extreme values of the response is reduced. In numerical examples and a real data application, it performs well.

A new reproducing kernel‐based nonlinear dimension reduction method for survival data

AbstractBased on the theories of sliced inverse regression (SIR) and reproducing kernel Hilbert space (RKHS), a new approach RDSIR (RKHS‐based Double SIR) to nonlinear dimension reduction for survival data is proposed. An isometric isomorphism is constructed based on the RKHS property, then the nonlinear function in the RKHS can be represented by the inner product of two elements that reside in the isomorphic feature space. Due to the censorship of survival data, double slicing is used to estimate the weight function to adjust for the censoring bias. The nonlinear sufficient dimension reduction (SDR) subspace is estimated by a generalized eigen‐decomposition problem. The asymptotic property of the estimator is established based on the perturbation theory. Finally, the performance of RDSIR is illustrated on simulated and real data. The numerical results show that RDSIR is comparable with the linear SDR method. Most importantly, RDSIR can also effectively extract nonlinearity from survival data.

On a nonlinear extension of the principal fitted component model

We propose a nonlinear sufficient dimension reduction method called the kernel principal fitted component model using the kernel method under a reproducing kernel Hilbert space. The kernel principal fitted component is a nonlinear extension of the principal fitted component model, and it is found in the theory of mapping low dimensional input space to the higher dimensional feature space so that we can apply well-developed linear methods to the nonlinear dataset. We derive our method coincides with the generalized sliced inverse regression under some mild assumptions and show the dimension reduction subspace extracted from the kernel principal fitted component model is contained in the central class. In the numerical experiments, we present the kernel principal fitted component model with the Gaussian kernel can extract the linear and nonlinear features well for the models from both forward and inverse regression settings. By applying our method to ovarian cancer microarray dataset, we demonstrate the kernel principal fitted component can provide a competitive prediction accuracy and computational efficiency in the high-dimensional classification problem.

Inference for the dimension of a regression relationship using pseudo-covariates.

In data analysis using dimension reduction methods, the main goal is to summarize how the response is related to the covariates through a few linear combinations. One key issue is to determine the number of independent, relevant covariate combinations, which is the dimension of the sufficient dimension reduction (SDR) subspace. In this work, we propose an easily-applied approach to conduct inference for the dimension of the SDR subspace, based on augmentation of the covariate set with simulated pseudo-covariates. Applying the partitioning principal to the possible dimensions, we use rigorous sequential testing to select the dimensionality, by comparing the strength of the signal arising from the actual covariates to that appearing to arise from the pseudo-covariates. We show that under a "uniform direction" condition, our approach can be used in conjunction with several popular SDR methods, including sliced inverse regression. In these settings, the test statistic asymptotically follows a beta distribution and therefore is easily calibrated. Moreover, the family-wise type I error rate of our sequential testing is rigorously controlled. Simulation studies and an analysis of newborn anthropometric data demonstrate the robustness of the proposed approach, and indicate that the power is comparable to or greater than thealternatives.

Online sparse sliced inverse regression for high-dimensional streaming data

Due to the demand for tackling the problem of streaming data with high-dimensional covarites, we propose an online sparse sliced inverse regression (OSSIR) method for online sufficient dimension reduction (SDR). The existing online SDR methods focus on the case when [Formula: see text] (dimension of covariates) is small. In this paper, we adapt the sparse sliced inverse regression to cope with high-dimensional streaming data where the dimension [Formula: see text] is large. There are two important steps in our method, one is to extend the online principal component analysis to iteratively obtain the eigenvalues and eigenvectors of the kernel matrix, the other is to use the truncated gradient to perform online [Formula: see text] regularization. Theoretical properties of the proposed online learner are established. By comparing with several existing methods in simulations and real data applications, we demonstrate the effectiveness and efficiency of our method.

Projection expectile regression for sufficient dimension reduction

Many existing sufficient dimension reduction methods are designed for regression with predictors that are elliptically distributed, which limits their application in real data analyses. Projection expectile regression (PER) is proposed as a new linear sufficient dimension reduction method for handling complex predictor structures, which includes continuous, discrete, and mixed predictor variables. PER requires the link function between the response and the predictor to be monotone, but not necessarily smooth, which makes it suitable for handling stratified response surfaces. By design, PER does not involve matrix inversion or high-dimensional smoothing. Thus, PER is ideal for controlling problems associated with multicollinearity, high dimensionality, and sparsity in the predictor. An extensive simulation study demonstrates the performance of projection expectile regression in synthetic data. A real data analysis of health insurance charges in the United States is also provided. The asymptotic properties of the PER estimator are included as well.

Subspace Estimation with Automatic Dimension and Variable Selection in Sufficient Dimension Reduction

Sufficient dimension reduction (SDR) methods target finding lower-dimensional representations of a multivariate predictor to preserve all the information about the conditional distribution of the response given the predictor. The reduction is commonly achieved by projecting the predictor onto a low-dimensional subspace. The smallest such subspace is known as the Central Subspace (CS) and is the key parameter of interest for most SDR methods. In this article, we propose a unified and flexible framework for estimating the CS in high dimensions. Our approach generalizes a wide range of model-based and model-free SDR methods to high-dimensional settings, where the CS is assumed to involve only a subset of the predictors. We formulate the problem as a quadratic convex optimization so that the global solution is feasible. The proposed estimation procedure simultaneously achieves the structural dimension selection and coordinate-independent variable selection of the CS. Theoretically, our method achieves dimension selection, variable selection, and subspace estimation consistency at a high convergence rate under mild conditions. We demonstrate the effectiveness and efficiency of our method with extensive simulation studies and real data examples. Supplementary materials for this article are available online.

Conditional variance estimator for sufficient dimension reduction

Conditional Variance Estimation (CVE) is a novel sufficient dimension reduction (SDR) method for additive error regressions with continuous predictors and link function. It operates under the assumption that the predictors can be replaced by a lower dimensional projection without loss of information. Conditional Variance Estimation is fully data driven, does not require the restrictive linearity and constant variance conditions, and is not based on inverse regression as the majority of moment and likelihood based sufficient dimension reduction methods. CVE is shown to be consistent and its objective function to be uniformly convergent. CVE outperforms the mean average variance estimation, (MAVE), its main competitor, in several simulation settings, remains on par under others, while it always outperforms inverse regression based linear SDR methods, such as Sliced Inverse Regression.

On cross-distance selection algorithm for hybrid sufficient dimension reduction

Given the extensive development of a variety of sufficient dimension reduction (SDR) methodologies, Ye and Weiss (2003) proposed a hybrid SDR method combining two pre-existing SDR methods. In particular, they used a bootstrap approach to select a proper weight. Since bootstrapping is computationally intensive and time-consuming, the hybrid reduction approach has not been widely used, although it is more accurate than conventional single SDR methods. To overcome these deficits, we propose a novel cross-distance selection algorithm. Similar to the bootstrapping method, the proposed selection algorithm is data-driven and has a strong rationale for its performance. The numerical studies demonstrate that the chosen hybrid method from our proposed algorithm offers a good estimation quality and reduces the computing time dramatically at the same time. Furthermore, our real data analysis confirms that the proposed selection algorithm has potential advantages with its practical usefulness over the existing bootstrapping method.

Functional sufficient dimension reduction based on weighted method

In the last 20 years, many of functional sufficient dimension reduction (FSDR) methods are developed. Among all of the FSDR methods, the most popular methods are functional sliced inverse regression (FSIR) and functional sliced average variance estimation (FSAVE). FSIR and FSAVE are based on the first and second moment of the functional predictor, which are not robust to outlier. A robust version of FSIR called the soft weighted FSIR (SFSIR) is proposed method, which used a trimmed and spatial median estimate to replace the classic moment estimation. The trimmed estimate is very useful and SFSIR method can deal with the outlier data very well. However, discarding the potential outlier usually leads to low efficient estimate. And the trimmed estimation method need to select two tuning parameters and to detect the potential outlier is very computational costing. Furthermore, FSIR and SFSIR methods are well-known for their poor performance when the link function is even (or symmetrically dependency). Then, a robust FSDR method with high efficient and low computation costing estimate is urgent needed in practical. In the presented paper, firstly, we will propose weighted based FSIR and FASVE methods. Secondly, we will also study the robust FSIR and FSAVE methods by series expansion, which first approximates the functional predictor by functional principal component basis, and then conduct FSDR methods on the coefficients. Numerical studies including simulation study and real data analysis show that our proposed weighted methods have wonderful performance.

Model averaging‐based sufficient dimension reduction

Sufficient dimension reduction is intended to project high‐dimensional predictors onto a low‐dimensional space without loss of information on the responses. Classical methods, such as sliced inverse regression, sliced average variance estimation and directional regression, are backbones of many modern sufficient dimension methods and have gained considerable research interests. However, the efficiency of such methods will be shrunk when dealing with sparse models. Under given models or some strict sparsity assumptions, there are existing sparse sufficient dimension methods in the literature. In order to relax the model assumptions and sparsity, in this paper, we define a general least squares objective function, which is applicable to all kernel matrices of classical sufficient dimension reduction methods, and propose a Mallows model averaging based sufficient dimension reduction method. Furthermore, an iterative least squares algorithm is used to obtain the sample estimates. Our method demonstrates excellent performance in simulation results.

Missing data analysis with sufficient dimension reduction

AbstractThis article develops a two‐step procedure for estimating the unknown parameters in a model that contains a fixed but large number of covariates, more moment conditions than unknown parameters, and responses that are missing at random. We propose a sufficient dimension reduction method to be implemented in the first step and prove that the method is asymptotically valid. In the second step, we apply three well‐known missing data handling mechanisms together with the generalized method of moments to the reduced‐dimensional subspace to obtain estimates of unknown parameters. We investigate the theoretical properties of the proposed methods, including the effects of dimension reduction on the asymptotic distributions of the estimators. Our results refute a claim in an earlier study that dimension reduction yields the same asymptotic distributions of estimators as when the reduced‐dimensional structure is the true structure. We illustrate our method by way of a simulation study and a real clinical trial data example.

FUNCTIONAL SUFFICIENT DIMENSION REDUCTION THROUGH AVERAGE FRÉCHET DERIVATIVES.

Sufficient dimension reduction (SDR) embodies a family of methods that aim for reduction of dimensionality without loss of information in a regression setting. In this article, we propose a new method for nonparametric function-on-function SDR, where both the response and the predictor are a function. We first develop the notions of functional central mean subspace and functional central subspace, which form the population targets of our functional SDR. We then introduce an average Fréchet derivative estimator, which extends the gradient of the regression function to the operator level and enables us to develop estimators for our functional dimension reduction spaces. We show the resulting functional SDR estimators are unbiased and exhaustive, and more importantly, without imposing any distributional assumptions such as the linearity or the constant variance conditions that are commonly imposed by all existing functional SDR methods. We establish the uniform convergence of the estimators for the functional dimension reduction spaces, while allowing both the number of Karhunen-Loève expansions and the intrinsic dimension to diverge with the sample size. We demonstrate the efficacy of the proposed methods through both simulations and two real data examples.

Distributed estimation in heterogeneous reduced rank regression: With application to order determination in sufficient dimension reduction

We are concerned with massive data which are possibly heterogeneous and scattered at different locations. We introduce a communication-efficient distributed algorithm to estimate the rank-deficient loading matrix in reduced rank regressions. The distributed algorithm, which proceeds iteratively, reduces the computational complexity substantially. During each iteration, it yields a closed-form solution and refines the previous estimators gradually. After a finite number of iterations, the final solution estimates the rank consistently, and more importantly, achieves the oracle rate. We recast sufficient dimension reduction methods under the framework of reduced rank regressions, which enables us to recover the central subspace and simultaneously estimate its structural dimension. We demonstrate the efficiency of our proposed distributed algorithm through simulations and an application to the airline on-line performance dataset consisting of 118,914,458 observations.

Local influence analysis for the sliced average third‐moment estimation

AbstractSliced average third‐moment estimation (SATME) is a typical method for sufficient dimension reduction (SDR) based on high‐order conditional moment. It is useful, particularly in the scenarios of regression mixtures. However, as SATME uses the third‐order conditional moment of the predictors given the response, it may not as robust as some other SDR methods that use lower order moments, say, sliced inverse regression (SIR) and slice average variance estimation (SAVE). Based on the space displacement function, a local influence analysis framework of SATME is constructed including a statistic of influence assessment for the observations. Furthermore, a data‐trimming strategy is suggested based on the above influence assessment. The proposed methodologies solve a typical issue that also exists in some other SDR methods. A real‐data analysis and simulations are presented.

Sufficient Dimension Reduction: An Information-Theoretic Viewpoint.

There has been a lot of interest in sufficient dimension reduction (SDR) methodologies, as well as nonlinear extensions in the statistics literature. The SDR methodology has previously been motivated by several considerations: (a) finding data-driven subspaces that capture the essential facets of regression relationships; (b) analyzing data in a ‘model-free’ manner. In this article, we develop an approach to interpreting SDR techniques using information theory. Such a framework leads to a more assumption-lean understanding of what SDR methods do and also allows for some connections to results in the information theory literature.

The ensemble conditional variance estimator for sufficient dimension reduction

Ensemble Conditional Variance Estimation (ECVE) is a novel sufficient dimension reduction (SDR) method in regressions with continuous response and predictors. ECVE applies to general non-additive error regression models and operates under the assumption that the predictors can be replaced by a lower dimensional projection without loss of information. It is a semiparametric forward regression model-based exhaustive sufficient dimension reduction estimation method that is shown to be consistent under mild assumptions. ECVE outperforms central subspace mean average variance estimation (csMAVE), its main competitor, under several simulation settings and in a benchmark data set analysis.