Finite Information Geometry

Abstract

This chapter investigates probability distributions on a finite sample space and takes advantage of the more elementary nature of this setting. There are two complementary ways to view a probability distribution. One consists in viewing it as (positive) measure with total mass 1. The other considers it as an equivalence class of such measures, determined up to a global scaling factor. The natural geometry underlying the first is that of the unit simplex (probability simplex). The second leads to the projective space of positive measures, which here simply is the positive sector of the unit sphere. The Euclidean metric induces a metric on the sphere, and that is the Fisher metric. In contrast, the simplex carries a natural flat structure. With respect to the Fisher metric, we get a dual flat structure. These two flat structures are encoded by the Amari–Chentsov tensor. This chapter then explores the characteristic properties of the Fisher and Amari–Chentsov tensors for finite sample spaces, setting the stage for corresponding results for general sample spaces in subsequent chapters. It also introduces divergences and exponential and mixture families of probability distributions and describes the Pythagorean geometry of projections onto such families. Finally, the geometry of graphical and hierarchical models is analyzed with those tools.