A formula of equations of states in singular learning machines

Abstract

Almost all learning machines used in computational intelligence are not regular but singular statistical models, because they are nonidentifiable and their Fisher information matrices are singular. In singular learning machines, neither the Bayes a posteriori distribution converges to the normal distribution nor the maximum likelihood estimator satisfies the asymptotic normality, resulting that it has been difficult to estimate generalization performances. In this paper, we establish a formula of equations of states which holds among Bayes and Gibbs generalization and training errors, and show that two generalization errors can be estimated from two training errors. The equations of states proved in this paper hold for any true distribution, any learning machine, and a priori distribution, and any singularities, hence they define widely applicable information criteria.