Effects of priors in model selection problem of learning machines with singularities

Abstract

AbstractIn statistical learning by information systems, operations are required for selecting the model that best fits the given criterion from among multiple model candidates. The types of criteria that can be used include a consistency criterion for detecting the true distribution according to a maximum probability and an efficiency criterion for minimizing the prediction error. Although criteria such as the AIC or BIC have been proposed and their characteristics have been studied in detail for a regular statistical model, there are many unknowns concerning criteria for selecting a learning model having singularities such as a neural network or mixed normal distribution. In this paper, the authors examine Bayesian learning by models having singularities and compare a method that always uses a positive prior distribution and a method that uses Jeffreys' prior distribution from viewpoints related to consistency and efficiency when selecting a model for minimizing the stochastic complexity. Based on a theoretical proposition that is already known, they rationally predict the difference between the two distributions and verify that prediction experimentally. In particular, they clearly demonstrate experimentally that when the family of learning models includes the true distribution, Jeffreys' prior distribution is superior in terms of both consistency and efficiency, but when the family of learning models does not include the true distribution, the prior distribution that always takes positive values is superior in terms of efficiency. © 2005 Wiley Periodicals, Inc. Electron Comm Jpn Pt 2, 88(2): 47–58, 2005; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecjb.20147