High‐dimensional regression coefficient estimation by nuclear norm plus l1 norm penalization

Abstract

We propose a new estimator of the regression coefficients for a high‐dimensional linear regression model, which is derived by replacing the sample predictor covariance matrix in the ordinary least square (OLS) estimator with a different predictor covariance matrix estimate obtained by a nuclear norm plus norm penalization. We call the estimator ALgebraic Covariance Estimator‐regression (ALCE‐reg). We make a direct theoretical comparison of the expected mean square error of ALCE‐reg with OLS and RIDGE. We show in a simulation study that ALCE‐reg is particularly effective when both the dimension and the sample size are large, due to its ability to find a good compromise between the large bias of shrinkage estimators (like RIDGE and least absolute shrinkage and selection operator [LASSO]) and the large variance of estimators conditioned by the sample predictor covariance matrix (like OLS and principal orthogonal complement thresholding [POET]).