Dimensional Covariance Research Articles

Adjusted regularized estimation in the accelerated failure time model with high dimensional covariates

Based on Stute’s weighted least squares method, we consider the estimate procedures for the accelerated failure time (AFT) model with high dimensional covariates. We use Kaplan–Meier weights and Stute’s estimator to account for censoring in least squares estimation. We consider two regularization approaches, the least absolute shrinkage and selection operator (LASSO) and minimax concave penalty (MCP) for estimation and variable selection in the AFT model. Our work includes the following contributions: (1) we adjust the penalty term in the objective function and show that this adjustment is crucial for the AFT model when the scales of covariates are heterogeneous; (2) with this adjustment, we also get a desired property for LASSO estimator in the AFT model, just like the uncensored case; (3) we replace a key assumption required in the high dimensional AFT model with a set of conditions which essentially require the fraction of censored data to be small; (4) we get selection consistency of MCP in the AFT model.

Confidence Intervals for Low Dimensional Parameters in High Dimensional Linear Models

SummaryThe purpose of this paper is to propose methodologies for statistical inference of low dimensional parameters with high dimensional data. We focus on constructing confidence intervals for individual coefficients and linear combinations of several of them in a linear regression model, although our ideas are applicable in a much broader context. The theoretical results that are presented provide sufficient conditions for the asymptotic normality of the proposed estimators along with a consistent estimator for their finite dimensional covariance matrices. These sufficient conditions allow the number of variables to exceed the sample size and the presence of many small non-zero coefficients. Our methods and theory apply to interval estimation of a preconceived regression coefficient or contrast as well as simultaneous interval estimation of many regression coefficients. Moreover, the method proposed turns the regression data into an approximate Gaussian sequence of point estimators of individual regression coefficients, which can be used to select variables after proper thresholding. The simulation results that are presented demonstrate the accuracy of the coverage probability of the confidence intervals proposed as well as other desirable properties, strongly supporting the theoretical results.

Empirical and weighted conditional likelihoods for matched case-control studies with missing covariates

In clinical and epidemiological studies, matched case-control designs have been used extensively to investigate the relationships between disease/response and exposure/covariate. Due to the retrospective nature of the study, some covariates may not be observed for all study subjects and missing covariate information may create bias and reduce the efficiency of the parameter estimates. We explore the use of profile empirical likelihood (EL) to cope with this situation by combining unbiased estimating equations when the number of estimating equations is greater than the number of unknown parameters. For high dimensional covariates, we propose a weighted conditional likelihood (WCL) method to solve the computational problem of the profile EL method. The proposed EL and WCL methods can achieve semiparametric efficiency if the probability of missingness is correctly specified. Based on the EL and WCL functions, we also develop Wilks’ type tests and corresponding confidence regions for the model parameters. A simulation study is conducted to assess the performance of the proposed methods in terms of robustness and efficiency.

Best permutation analysis

High dimensional covariance estimation is an important topic in contemporary multivariate statistics and has recently received much attention in the mathematical statistics literature. The work of Bickel and Levina (2008) [2] introduces a general approach to such estimation problems in a large class of models: banding of the sample covariance matrix. Bickel and Levina show that banded estimators are consistent in the operator norm as the dimension of the covariance matrix, p, and the sample size, n, both go to infinity. Critically, these estimators rely on knowing the order of the covariates apriori before banding can be applied. A rigorous framework for order recovery is however not available in the literature. In this paper, we propose a novel framework and methodology that can be used to recover covariate order in general classes of banded models. Such models can also be framed as autoregressive processes, which in turn fall within the class of graphical models. We show that recovering covariate order is intimately related to minimizing functionals on the symmetric group. Indeed, an important contribution of the paper is a result showing that the natural time order in such an autoregressive process has the property that over all orderings of covariates, it minimizes the sum of the diagonals of the Cholesky decomposition, of both the covariance and the inverse covariance matrix. This result lays the foundation for the ensuing statistical methodology developed in this paper: an efficient algorithm called the Best Permutation Algorithm (BPA). The BPA can recover the natural order of variables in autoregressive models at the rate of Op((logp)/n), which is the same rate that the covariance matrix can be estimated if the natural time order were known. Hence the BPA yields the oracle rate. Moreover, the computational complexity of the BPA is proved to be polynomial in the number of variables, p, and hence allows for an efficient search over the full permutation group on p letters, a group whose size is super-exponential in p. The methodology is also successfully illustrated on numerical examples.

Long memory conditional volatility and asset allocation

In this paper, we evaluate the economic benefits that arise from allowing for long memory when forecasting the covariance matrix of returns over both short and long horizons, using the asset allocation framework of Engle and Colacito (2006) In particular, we compare the statistical and economic performances of four multivariate long memory volatility models (the long memory EWMA, long memory EWMA–DCC, FIGARCH-DCC and component GARCH-DCC models) with those of two short memory models (the short memory EWMA and GARCH-DCC models). We report two main findings. First, for longer horizon forecasts, long memory models generally produce forecasts of the covariance matrix that are statistically more accurate and informative, and economically more useful than those produced by short memory models. Second, the two parsimonious long memory EWMA models outperform the other models–both short and long memory–across most forecast horizons. These results apply to both low and high dimensional covariance matrices and both low and high correlation assets, and are robust to the choice of the estimation window.

Remodeling and Estimation for Sparse Partially Linear Regression Models

When the dimension of covariates in the regression model is high, one usually uses a submodel as a working model that contains significant variables. But it may be highly biased and the resulting estimator of the parameter of interest may be very poor when the coefficients of removed variables are not exactly zero. In this paper, based on the selected submodel, we introduce a two-stage remodeling method to get the consistent estimator for the parameter of interest. More precisely, in the first stage, by a multistep adjustment, we reconstruct an unbiased model based on the correlation information between the covariates; in the second stage, we further reduce the adjusted model by a semiparametric variable selection method and get a new estimator of the parameter of interest simultaneously. Its convergence rate and asymptotic normality are also obtained. The simulation results further illustrate that the new estimator outperforms those obtained by the submodel and the full model in the sense of mean square errors of point estimation and mean square prediction errors of model prediction.

A Review on Dimension Reduction

Summarizing the effect of many covariates through a few linear combinations is an effective way of reducing covariate dimension and is the backbone of (sufficient) dimension reduction. Because the replacement of high-dimensional covariates by low-dimensional linear combinations is performed with a minimum assumption on the specific regression form, it enjoys attractive advantages as well as encounters unique challenges in comparison with the variable selection approach. We review the current literature of dimension reduction with an emphasis on the two most popular models, where the dimension reduction affects the conditional distribution and the conditional mean, respectively. We discuss various estimation and inference procedures in different levels of detail, with the intention of focusing on their underneath idea instead of technicalities. We also discuss some unsolved problems in this area for potential future research.

Condition-Number-Regularized Covariance Estimation

Estimation of high-dimensional covariance matrices is known to be a difficult problem, has many applications, and is of current interest to the larger statistics community. In many applications including so-called the "large p small n" setting, the estimate of the covariance matrix is required to be not only invertible, but also well-conditioned. Although many regularization schemes attempt to do this, none of them address the ill-conditioning problem directly. In this paper, we propose a maximum likelihood approach, with the direct goal of obtaining a well-conditioned estimator. No sparsity assumption on either the covariance matrix or its inverse are are imposed, thus making our procedure more widely applicable. We demonstrate that the proposed regularization scheme is computationally efficient, yields a type of Steinian shrinkage estimator, and has a natural Bayesian interpretation. We investigate the theoretical properties of the regularized covariance estimator comprehensively, including its regularization path, and proceed to develop an approach that adaptively determines the level of regularization that is required. Finally, we demonstrate the performance of the regularized estimator in decision-theoretic comparisons and in the financial portfolio optimization setting. The proposed approach has desirable properties, and can serve as a competitive procedure, especially when the sample size is small and when a well-conditioned estimator is required.

Doubly Robust and Efficient Estimators for Heteroscedastic Partially Linear Single-Index Models Allowing high Dimensional Covariates

We study the heteroscedastic partially linear single-index model with an unspecified error variance function, which allows for high dimensional covariates in both the linear and the single-index components of the mean function. We propose a class of consistent estimators of the parameters by using a proper weighting strategy. An interesting finding is that the linearity condition which is widely assumed in the dimension reduction literature is not necessary for methodological or theoretical development: it contributes only to the simplification of non-optimal consistent estimation. We also find that the performance of the usual weighted least square type of estimators deteriorates when the non-parametric component is badly estimated. However, estimators in our family automatically provide protection against such deterioration, in that the consistency can be achieved even if the baseline non-parametric function is completely misspecified. We further show that the most efficient estimator is a member of this family and can be easily obtained by using non-parametric estimation. Properties of the estimators proposed are presented through theoretical illustration and numerical simulations. An example on gender discrimination is used to demonstrate and to compare the practical performance of the estimators.

Mann–Whitney Test with Adjustments to Pretreatment Variables for Missing Values and Observational Study

SummaryThe conventional Wilcoxon or Mann–Whitney test can be invalid for comparing treatment effects in the presence of missing values or in observational studies. This is because the missingness of the outcomes or the participation in the treatments may depend on certain pretreatment variables. We propose an approach to adjust the Mann–Whitney test by correcting the potential bias via consistently estimating the conditional distributions of the outcomes given the pretreatment variables. We also propose semiparametric extensions of the adjusted Mann–Whitney test which lead to dimension reduction for high dimensional covariates. A novel bootstrap procedure is devised to approximate the null distribution of the test statistics for practical implementations. Results from simulation studies and an economics observational study data analysis are presented to demonstrate the performance of the approach proposed.

Learned-loss boosting

This paper considers a problem of jointly estimating a regression function and the distribution of residuals when both are specified non-parametrically. We present a joint penalized optimization criterion that combines log-spline density estimation with spline-based regression methods. We also examine the use of boosting methodology to estimate a regression function over a high dimensional covariate space. We demonstrate that our method has a robustification effect, and show its usefulness in diagnosing problems in data. We illustrate our methods with practical examples when likelihood is an appropriate evaluation criterion.

Consistent model selection criteria on high dimensions

Asymptotic properties of model selection criteria for high-dimensional regression models are studied where the dimension of covariates is much larger than the sample size. Several sufficient condit...

Profiled adaptive Elastic-Net procedure for partially linear models with high-dimensional covariates

We study variable selection for partially linear models when the dimension of covariates diverges with the sample size. We combine the ideas of profiling and adaptive Elastic-Net. The resulting procedure has oracle properties and can handle collinearity well. A by-product is the uniform bound for the absolute difference between the profiled and original predictors. We further examine finite sample performance of the proposed procedure by simulation studies and analysis of a labor-market dataset for an illustration.

Forward adaptive banding for estimating large covariance matrices

We propose a simple forward adaptive banding method for estimating large covariance matrices using the modified Cholesky decomposition. This approach requires the fitting of a prespecified set of models due to the adaptive banding structure and can be efficiently implemented. Aside from its computational attractiveness, we propose a novel Bayes information criterion that gives consistent model selection for estimating high dimensional covariance matrices. The method compares favourably to its competitors in simulation study.

Long Memory Conditional Volatility and Assett Allocation

In this paper, we evaluate the economic benefits that arise from allowing for long memory in forecasting the covariance matrix of returns over both short and long horizons, using the asset allocation framework of Engle and Colacito (2006). In particular, we compare the statistical and economic performance of four multivariate long memory volatility models (the long memory EWMA, long memory EWMA-DCC, FIGARCH-DCC and component GARCH-DCC models) with that of two short memory models (the short memory EWMA and GARCH-DCC models). We report two main findings. First, for longer horizon forecasts, long memory models produce forecasts of the covariance matrix that are statistically more accurate and informative, and economically more useful than those produced by short memory models. Second, the two parsimonious long memory EWMA models outperform the other models – both short memory and long memory – at all forecast horizons. These results apply to both low and high dimensional covariance matrices and both low and high correlation assets, and are robust to the choice of estimation window.

Image Dimensionality Reduction Based on the Intrinsic Dimension and Parallel Genetic Algorithm

In the research of Web content-based image retrieval, how to reduce more of the image dimensions without losing the main features of the image is highlighted. Many features of dimensional reduction schemes are determined by the breaking of higher dimensional general covariance associated with the selection of a particular subset of coordinates. This paper starts with analysis of commonly used methods for the dimension reduction of Web images, followed by a new algorithm for nonlinear dimensionality reduction based on the HSV image features. The approach obtains intrinsic dimension estimation by similarity calculation of two images. Finally, some improvements were made on the Parallel Genetic Algorithm (APGA) by use of the image similarity function as the self-adaptive judgment function to improve the genetic operators, thus achieving a Web image dimensionality reduction and similarity retrieval. Experimental results illustrate the validity of the algorithm.

On limiting spectral distribution of large sample covariance matrices by VARMA(p,q)

We studied the limiting spectral distribution of large-dimensional sample covariance matrices of a stationary and invertible VARMA(p,q) model. Relationship of the power spectral density and limiting spectral distribution of large population dimensional covariance matrices of ARMA(p,q) is established. The equation about Stieltjes transform of large-dimensional sample covariance matrices is also derived. As applications, the classical M-P law, VAR(1) and VMA(1) can be regarded as special examples.

High Dimensional Structured Additive Regression Models: Bayesian Regularization, Smoothing and Predictive Performance

SummaryData structures in modern applications frequently combine the necessity of flexible regression techniques handling, for example, non-linear and spatial effects with high dimensional covariate vectors. Whereas estimation of the former is typically achieved by supplementing the likelihood with a suitable smoothness penalty, the latter are usually assigned shrinkage penalties that enforce sparse models. We consider a Bayesian unifying perspective, where conditionally Gaussian priors can be assigned to all types of regression effects. Suitable hyperprior assumptions on the variances of the Gaussian distributions then induce the desired smoothness or sparseness properties. As a major advantage, general Markov chain Monte Carlo simulation algorithms can be developed that allow for the joint estimation of smooth and spatial effects and regularized coefficient vectors. Two applications demonstrate the usefulness of the procedure proposed: a geoadditive regression model for data from a rental guide in Munich and an additive probit model for the prediction of consumer credit defaults. In both cases, high dimensional vectors of categorical covariates will be included in the regression models. The predictive ability of the resulting high dimensional structured additive regression models compared with expert models is of particular relevance and will be evaluated on cross-validation test data.

Robust Shrinkage Estimation of High-Dimensional Covariance Matrices

We address high dimensional covariance estimation for elliptical distributed samples, which are also known as spherically invariant random vectors (SIRV) or compound-Gaussian processes. Specifically we consider shrinkage methods that are suitable for high dimensional problems with a small number of samples (large $p$ small $n$). We start from a classical robust covariance estimator [Tyler(1987)], which is distribution-free within the family of elliptical distribution but inapplicable when $n<p$. Using a shrinkage coefficient, we regularize Tyler's fixed point iterations. We prove that, for all $n$ and $p$, the proposed fixed point iterations converge to a unique limit regardless of the initial condition. Next, we propose a simple, closed-form and data dependent choice for the shrinkage coefficient, which is based on a minimum mean squared error framework. Simulations demonstrate that the proposed method achieves low estimation error and is robust to heavy-tailed samples. Finally, as a real world application we demonstrate the performance of the proposed technique in the context of activity/intrusion detection using a wireless sensor network.

Combining conditional and unconditional moment restrictions with missing responses

Many statistical models, e.g. regression models, can be viewed as conditional moment restrictions when distributional assumptions on the error term are not assumed. For such models, several estimators that achieve the semiparametric efficiency bound have been proposed. However, in many studies, auxiliary information is available as unconditional moment restrictions. Meanwhile, we also consider the presence of missing responses. We propose the combined empirical likelihood (CEL) estimator to incorporate such auxiliary information to improve the estimation efficiency of the conditional moment restriction models. We show that, when assuming responses are strongly ignorable missing at random, the CEL estimator achieves better efficiency than the previous estimators due to utilization of the auxiliary information. Based on the asymptotic property of the CEL estimator, we also develop Wilks’ type tests and corresponding confidence regions for the model parameter and the mean response. Since kernel smoothing is used, the CEL method may have difficulty for problems with high dimensional covariates. In such situations, we propose an instrumental variable-based empirical likelihood (IVEL) method to handle this problem. The merit of the CEL and IVEL are further illustrated through simulation studies.