Comparing Penalized Regression Analysis of Logistic Regression Model with Multicollinearity

Abstract

The goal of this research is to estimate the parameter of the logistic regression model by penalized regression analysis which consisted of ridge regression, lasso, and elastic net method. The logistic regression is considered between a binary dependent variable and 3 and 5 independent variables. The independent variables are generated from normal distribution, contaminated normal distribution, and t distribution on correlation coefficient at 0.1, 0.5, and 0.99 or called multicollinearity problem. The maximum likelihood estimator has used as the classical method by differential the log likelihood function with respect to the coefficients. Ridge regression is to choose the unknown ridge parameter by cross-validation, so ridge estimator is evaluated by the adding ridge parameter on penalty term. Lasso (least absolute shrinkage and selection operator) is added the penalty term on scales sum of the absolute value of the coefficients. The elastic net can be mixed between ridge regression and lasso on the penalty term. The criterion of these methods is compared by percentage of predicted accuracy value. The results are found that lasso is satisfied when the independent variables are simulated from normal and t distribution in most cases, and the lasso outperforms on the contaminated normal distribution.