Pseudo estimation and variable selection in regression

Abstract

Often a problem in linear regression either for correlated data or high-dimensional data is the singularity of the sample covariance matrix. While dealing with an ill-conditioned covariance matrix has been a longstanding challenge in the statistical literature, recent proposals relying on adding noises to the original data have found their successes in obtaining reliable estimates and making predictions. It has been shown that perturbing data with noises has a close relationship to the well-known ridge estimator. In this paper, we propose to add noises to the predictors with a known covariance structure, and call the estimator obtained in such a way a “pseudo estimator”. A new variable selection procedure based on the concept of pseudo confidence interval which works for both correlated predictors and “large p small n” problem is proposed. We study the theoretical properties of the proposed pseudo estimator and variable selection procedure. Furthermore, we use an ensemble step to stabilize the pseudo estimation (variable selection) results. The advantages of the proposed method are demonstrated by both simulation studies and real data analyses.