Pythagoras theorem in information geometry and applications to generalized linear models

Abstract

Abstract We have a statistical introduction for Information Geometry focusing the Pythagoras theorem in a space of probability density or mass functions when the squared length is defined by the Kullback–Leibler divergence. It is reviewed that the Pythagoras theorem extends to a foliation structure of the subspace associated with the maximum likelihood estimator (MLE) under the assumption of an exponential model. We discuss such a perspective in a framework of regression model. A simple example of the Pythagoras theorem comes out the Gauss least squares in a linear regression model, in which the assumption of a normal distribution is strongly connected with the MLE. On the other hand, we consider another estimator called the minimum power estimator. We extend the couple of the normal distribution model and the MLE to another couple of the t-distribution model and the minimum power estimator, which exactly associate with the dualistic structure if the power defining the estimator is matched by the degrees of freedom of the t-distribution. These observations can be applied in a framework of generalized linear model. Under an exponential-dispersion model including the Bernoulli, Poisson, and exponential distributions, the MLE leads to the Pythagoras foliation, which reveals the decomposition of the deviance statistics. This is parallel to the discussion for the residual sum of squares in the Pythagoras theorem.