Practical Treatment of the Multicollinearity: The Optimal Ridge Method and the Modified OLS

Abstract

The paper discusses the applicability of the two main methods for solving the linear regression (LR) problem in the presence of multicollinearity – the OLS and the ridge methods. We compare the solutions obtained by these methods with the solution calculated by the Modified OLS (MOLS) [1; 2]. Like the ridge, the MOLS provides a stable solution for any level of data collinearity. We compare three approaches by using the Monte Carlo simulations, and the data used is generated by the Artificial Data Generator (ADG) [1; 2]. The ADG produces linear and nonlinear data samples of arbitrary size, which allows the investigation of the OLS equation's regularization problem. Two possible regularization versions are the COV version considered in [1; 2] and the ST version commonly used in the literature and practice. The performed investigations reveal that the ridge method in the COV version has an approximately constant optimal regularizer (?_ridge^((opt))?0.1) for any sample size and collinearity level. The MOLS method in this version also has an approximately constant optimal regularizer, but its value is significantly smaller (?_MOLS^((opt))?0.001). On the contrary, the ridge method in the ST version has the optimal regularizer, which is not a constant but depends on the sample size. In this case, its value needs to be set to ?_ridge^((opt))?0.1(n-1). With such a value of the ridge parameter, the obtained solution is strictly the same as one obtained with the COV version but with the optimal regularizer ?_ridge^((opt))?0.1 [1; 2]. With such a choice of the regularizer, one can use any implementation of the ridge method in all known statistical software by setting the regularization parameter ?_ridge^((opt))?0.1(n-1) without extra tuning process regardless of the sample size and the collinearity level. Also, it is shown that such an optimal ridge(0.1) solution is close to the population solution for a sample size large enough, but, at the same time, it has some limitations. It is well known that the ridge(0.1) solution is biased. However, as it has been shown in the paper, the bias is economically insignificant. The more critical drawback, which is revealed, is the smoothing of the population solution – the ridge method significantly reduces the difference between the population regression coefficients. The ridge(0.1) method can result in a solution, which is economically correct, i.e., the regression coefficients have correct signs, but this solution might be inadequate to a certain extent. The more significant the difference between the regression coefficients in the population, the more inadequate is the ridge(0.1) method. As for the MOLS, it does not possess this disadvantage. Since its regularization constant is much smaller than the corresponding ridge regularizer (0.001 versus 0.1), the MOLS method suffers little from both the bias and smoothing of its solutions. From a practical point of view, both the ridge(0.1) and the MOLS methods result in close stable solutions to the LR problem for any sample size and collinearity level. With the sample size increasing, both solutions approach the population solution. We also demonstrate that for a small sample size of less than 40, the ridge(0.1) method is preferable, as it is more stable. When the sample size is medium or large, it is preferable to use the MOLS as it is more accurate yet has approximately the same stability.