Tuning Parameter Selection Methods Research Articles

Stability Approach to Regularization Selection for Reduced-Rank Regression

The reduced-rank regression model is a popular model to deal with multivariate response and multiple predictors, and is widely used in biology, chemometrics, econometrics, engineering, and other fields. In the reduced-rank regression modeling, a central objective is to estimate the rank of the coefficient matrix that represents the number of effective latent factors in predicting the multivariate response. Although theoretical results such as rank estimation consistency have been established for various methods, in practice rank determination still relies on information criterion based methods such as AIC and BIC or subsampling based methods such as cross-validation. Unfortunately, the theoretical properties of these practical methods are largely unknown. In this article, we present a novel method called StARS-RRR that selects the tuning parameter and then estimates the rank of the coefficient matrix for reduced-rank regression based on the stability approach. We prove that StARS-RRR achieves rank estimation consistency, that is, the rank estimated with the tuning parameter selected by StARS-RRR is consistent to the true rank. Through a simulation study, we show that StARS-RRR outperforms other tuning parameter selection methods including AIC, BIC, and cross-validation as it provides the most accurate estimated rank. In addition, when applied to a breast cancer dataset, StARS-RRR discovers a reasonable number of genetic pathways that affect the DNA copy number variations and results in a smaller prediction error than the other methods with a random-splitting process. Supplementary materials for this article are available online.

Globaltest confidence regions and their application to ridge regression.

We construct confidence regions in high dimensions by inverting the globaltest statistics, and use them to choose the tuning parameter for penalized regression. The selected model corresponds to the point in the confidence region of the parameters that minimizes the penalty, making it the least complex model that still has acceptable fit according to the test that defines the confidence region. As the globaltest is particularly powerful in the presence of many weak predictors, it connects well to ridge regression, and we thus focus on ridge penalties in this paper. The confidence region method is quick to calculate, intuitive, and gives decent predictive potential. As a tuning parameter selection method it may even outperform classical methods such as cross‐validation in terms of mean squared error of prediction, especially when the signal is weak. We illustrate the method for linear models in simulation study and for Cox models in real gene expression data of breast cancer samples.

Cross-Validation With Confidence

Cross-validation is one of the most popular model and tuning parameter selection methods in statistics and machine learning. Despite its wide applicability, traditional cross-validation methods tend to overfit, due to the ignorance of the uncertainty in the testing sample. We develop a novel statistically principled inference tool based on cross-validation that takes into account the uncertainty in the testing sample. This method outputs a set of highly competitive candidate models containing the optimal one with guaranteed probability. As a consequence, our method can achieve consistent variable selection in a classical linear regression setting, for which existing cross-validation methods require unconventional split ratios. When used for tuning parameter selection, the method can provide an alternative trade-off between prediction accuracy and model interpretability than existing variants of cross-validation. We demonstrate the performance of the proposed method in several simulated and real data examples. Supplemental materials for this article can be found online.

Tuning Parameter Selection Based on Blocked $$3\times 2$$ Cross-Validation for High-Dimensional Linear Regression Model

In high-dimensional linear regression, selecting an appropriate tuning parameter is essential for the penalized linear models. From the perspective of the expected prediction error of the model, cross-validation methods are commonly used to select the tuning parameter in machine learning. In this paper, blocked \(3\times 2\) cross-validation (\(3\times 2\) BCV) is proposed as the tuning parameter selection method because of its small variance for the prediction error estimation. Under some weaker conditions than leave-\(n_v\)-out cross-validation, the tuning parameter selection method based on \(3\times 2\) BCV is proved to be consistent for the high-dimensional linear regression model. Furthermore, simulated and real data experiments support the theoretical results and demonstrate that the proposed method works well in several criteria about selecting the true model.

Sentinel-2 Sharpening Using a Reduced-Rank Method

Recently, the Sentinel-2 (S2) satellite constellation was deployed for mapping and monitoring the Earth environment. Images acquired by the sensors mounted on the S2 platforms have three levels of spatial resolution: 10, 20, and 60 m. In many remote sensing applications, the availability of images at the highest spatial resolution (i.e., 10 m for S2) is often desirable. This can be achieved by generating a synthetic high-resolution image through data fusion. To this end, researchers have proposed techniques exploiting the spectral/spatial correlation inherent in multispectral data to sharpen the lower resolution S2 bands to 10 m. In this paper, we propose a novel method that formulates the sharpening process as a solution to an inverse problem. We develop a cyclic descent algorithm called S2Sharp and an associated tuning parameter selection algorithm based on generalized cross validation and Bayesian optimization. The tuning parameter selection method is evaluated on a simulated data set. The effectiveness of S2Sharp is assessed experimentally by comparisons to state-of-the-art methods using both simulated and real data sets.

Stability enhanced variable selection for a semiparametric model with flexible missingness mechanism and its application to the ChAMP study

ABSTRACT This paper is motivated by the analytical challenges we encounter when analyzing the ChAMP (Chondral Lesions And Meniscus Procedures) study, a randomized controlled trial to compare debridement to observation of chondral lesions in arthroscopic knee surgery. The main outcome, WOMAC (Western Ontario and McMaster Universities Osteoarthritis Index) pain score, is derived from the patient's responses to the questionnaire collected in the study. The major goal is to identify potentially important variables that contribute to this outcome. In this paper, the model of interest is a semiparametric model for the pain score. To address the missing data issue, we adopt a flexible missingness mechanism that is much more versatile in practice than a single parametric model. Then we propose a pairwise conditional likelihood approach to estimate the unknown parameter in the semiparametric model without the need of modeling its nonparametric counterpart nor the missingness mechanism. For variable selection, we apply a regularization approach with a variety of stability enhanced tuning parameter selection methods. We conduct comprehensive simulation studies to evaluate the performance of the proposed method. We also apply the proposed method to the ChAMP study to demonstrate its usefulness.

Tuning Parameter Selection in Cox Proportional Hazards Model with a Diverging Number of Parameters.

Regularized variable selection is a powerful tool for identifying the true regression model from a large number of candidates by applying penalties to the objective functions. The penalty functions typically involve a tuning parameter that control the complexity of the selected model. The ability of the regularized variable selection methods to identify the true model critically depends on the correct choice of the tuning parameter. In this study we develop a consistent tuning parameter selection method for regularized Cox's proportional hazards model with a diverging number of parameters. The tuning parameter is selected by minimizing the generalized information criterion. We prove that, for any penalty that possesses the oracle property, the proposed tuning parameter selection method identifies the true model with probability approaching one as sample size increases. Its finite sample performance is evaluated by simulations. Its practical use is demonstrated in the Cancer Genome Atlas (TCGA) breast cancer data.

An enhanced topologically significant directed random walk in cancer classification using gene expression datasets.

Microarray technology has become one of the elementary tools for researchers to study the genome of organisms. As the complexity and heterogeneity of cancer is being increasingly appreciated through genomic analysis, cancerous classification is an emerging important trend. Significant directed random walk is proposed as one of the cancerous classification approach which have higher sensitivity of risk gene prediction and higher accuracy of cancer classification. In this paper, the methodology and material used for the experiment are presented. Tuning parameter selection method and weight as parameter are applied in proposed approach. Gene expression dataset is used as the input datasets while pathway dataset is used to build a directed graph, as reference datasets, to complete the bias process in random walk approach. In addition, we demonstrate that our approach can improve sensitive predictions with higher accuracy and biological meaningful classification result. Comparison result takes place between significant directed random walk and directed random walk to show the improvement in term of sensitivity of prediction and accuracy of cancer classification.

A regularized variable selection procedure in additive hazards model with stratified case-cohort design

Case-cohort designs are commonly used in large epidemiological studies to reduce the cost associated with covariate measurement. In many such studies the number of covariates is very large. An efficient variable selection method is needed for case-cohort studies where the covariates are only observed in a subset of the sample. Current literature on this topic has been focused on the proportional hazards model. However, in many studies the additive hazards model is preferred over the proportional hazards model either because the proportional hazards assumption is violated or the additive hazards model provides more relevent information to the research question. Motivated by one such study, the Atherosclerosis Risk in Communities (ARIC) study, we investigate the properties of a regularized variable selection procedure in stratified case-cohort design under an additive hazards model with a diverging number of parameters. We establish the consistency and asymptotic normality of the penalized estimator and prove its oracle property. Simulation studies are conducted to assess the finite sample performance of the proposed method with a modified cross-validation tuning parameter selection methods. We apply the variable selection procedure to the ARIC study to demonstrate its practical use.

Marginal maximum likelihood estimation methods for the tuning parameters of ridge, power ridge, and generalized ridge regression

ABSTRACTThis study introduces fast marginal maximum likelihood (MML) algorithms for estimating the tuning (shrinkage) parameter(s) of the ridge and power ridge regression models, and an automatic plug-in MML estimator for the generalized ridge regression model, in a Bayesian framework. These methods are applicable to multicollinear or singular covariate design matrices, including matrices where the number of covariates exceeds the sample size. According to analyses of many real and simulated datasets, these MML-based ridge methods tend to compare favorably to other tuning parameter selection methods, in terms of computation speed, prediction accuracy, and ability to detect relevant covariates.

Variable selection and estimation using a continuous approximation to the $$L_0$$ L 0 penalty

Variable selection problems are typically addressed under the regularization framework. In this paper, an exponential type penalty which very closely resembles the \(L_0\) penalty is proposed, we called it EXP penalty. The EXP penalized least squares procedure is shown to consistently select the correct model and is asymptotically normal, provided the number of variables grows slower than the number of observations. EXP is efficiently implemented using a coordinate descent algorithm. Furthermore, we propose a modified BIC tuning parameter selection method for EXP and show that it consistently identifies the correct model, while allowing the number of variables to diverge. Simulation results and data example show that the EXP procedure performs very well in a variety of settings.

Variable selection for case-cohort studies with failure time outcome.

Case-cohort designs are widely used in large cohort studies to reduce the cost associated with covariate measurement. In many such studies the number of covariates is very large, so an efficient variable selection method is necessary. In this paper, we study the properties of a variable selection procedure using the smoothly clipped absolute deviation penalty in a case-cohort design with a diverging number of parameters. We establish the consistency and asymptotic normality of the maximum penalized pseudo-partial-likelihood estimator, and show that the proposed variable selection method is consistent and has an asymptotic oracle property. Simulation studies compare the finite-sample performance of the procedure with tuning parameter selection methods based on the Akaike information criterion and the Bayesian information criterion. We make recommendations for use of the proposed procedures in case-cohort studies, and apply them to the Busselton Health Study.

A Confidence Region Approach to Tuning for Variable Selection

We develop an approach to tuning of penalized regression variable selection methods by calculating the sparsest estimator contained in a confidence region of a specified level. Because confidence intervals/regions are generally understood, tuning penalized regression methods in this way is intuitive and more easily understood by scientists and practitioners. More importantly, our work shows that tuning to a fixed confidence level often performs better than tuning via the common methods based on Akaike information criterion (AIC), Bayesian information criterion (BIC), or cross-validation (CV) over a wide range of sample sizes and levels of sparsity. Additionally, we prove that by tuning with a sequence of confidence levels converging to one, asymptotic selection consistency is obtained, and with a simple two-stage procedure, an oracle property is achieved. The confidence-region-based tuning parameter is easily calculated using output from existing penalized regression computer packages. Our work also shows how to map any penalty parameter to a corresponding confidence coefficient. This mapping facilitates comparisons of tuning parameter selection methods such as AIC, BIC, and CV, and reveals that the resulting tuning parameters correspond to confidence levels that are extremely low, and can vary greatly across datasets. Supplemental materials for the article are available online.

Least squares approximation with a diverging number of parameters

Regularized regression with the ℓ 1 penalty is a popular approach for variable selection and coefficient estimation. For a unified treatment of the ℓ 1 -constrained model selection, Wang and Leng (2007) proposed the least squares approximation method (LSA) for a fixed dimension. LSA makes use of a quadratic expansion of the loss function and takes full advantage of the fast Lasso algorithm in Efron et al. (2004). In this paper, we extend the fixed dimension LSA to the situation with a diverging number of parameters. We show that LSA possesses the oracle properties under appropriate conditions when the number of variables grows with the sample size. We propose a new tuning parameter selection method which achieves the oracle properties. Extensive simulation studies confirmed the theoretical results.

Sparse principal component analysis via regularized low rank matrix approximation

Principal component analysis (PCA) is a widely used tool for data analysis and dimension reduction in applications throughout science and engineering. However, the principal components (PCs) can sometimes be difficult to interpret, because they are linear combinations of all the original variables. To facilitate interpretation, sparse PCA produces modified PCs with sparse loadings, i.e. loadings with very few non-zero elements. In this paper, we propose a new sparse PCA method, namely sparse PCA via regularized SVD (sPCA-rSVD). We use the connection of PCA with singular value decomposition (SVD) of the data matrix and extract the PCs through solving a low rank matrix approximation problem. Regularization penalties are introduced to the corresponding minimization problem to promote sparsity in PC loadings. An efficient iterative algorithm is proposed for computation. Two tuning parameter selection methods are discussed. Some theoretical results are established to justify the use of sPCA-rSVD when only the data covariance matrix is available. In addition, we give a modified definition of variance explained by the sparse PCs. The sPCA-rSVD provides a uniform treatment of both classical multivariate data and high-dimension-low-sample-size (HDLSS) data. Further understanding of sPCA-rSVD and some existing alternatives is gained through simulation studies and real data examples, which suggests that sPCA-rSVD provides competitive results.