An information approach to regularization parameter selection under modelmisspecification

Abstract

We review the information approach to regularization parameter selection and itsinformation complexity extension for the solution of discrete ill posedproblems. An information criterion for regularization parameter selectionwas first proposed by Shibata in the context of ridge regression as anextension of Takeuchi’s information criterion. In the information approach, theregularization parameter value is chosen to maximize the mean expected loglikelihood (MELL) of a model whose parameters are estimated usingthe maximum penalized likelihood method. Under the Gaussian noiseassumption such a choice coincides with the minimum of mean predictiveerror choice. Maximization of the MELL corresponds to minimization ofthe mean Kullback–Leibler information, that measures the deviation ofthe approximating (model) distribution from the true one. The resultingregularization parameter selection methods can handle possible functional anddistributional misspecifications when the usual assumptions of Gaussian noiseand/or linear relationship have been made but not met. We also suggest that inengineering applications it is beneficial to find ways of lowering the risk ofgetting grossly under-regularized solutions and that the new informationcomplexity regularization parameter selection method (RPSM) is one of thepossibilities. Several examples of applying the reviewed RPSMs are given.