Ridge regression optimization using a harmonious approach

Abstract

AbstractA critical component of ridge regression (RR) is determining the optimal ridge parameter value, λ, where λ≥0. Improper selection of λ not only generates an under‐ or overfitted model but also leads to incorrect conclusions in inter‐model comparison studies such as between RR, PLS, PCR and other modeling methods. Several methods for determining the optimal RR model are evaluated in this paper. For example, the commonly used ridge trace is identified as subjective and impractical. A direct calculation method from the literature yields over‐ or underfitted RR models with λ either too small or to large respectively. Methods for determining λ based on a harmonious approach are discussed. The harmonious approach optimizes λ by inspecting the bias/variance tradeoff. Of the methods investigated, plotting a variance indicator against a bias measure to yield an L‐curve appears not only to simplify selection of λ but also to reduce the chance of obtaining an under‐ or overfitted RR model. It is shown with four data sets that the L‐curve harmonious approach consistently provides good models. The effective rank of models is also discussed in conjunction with the harmony/parsimony tradeoff. Copyright © 2005 John Wiley & Sons, Ltd.