$\alpha$-Divergence Is Unique, Belonging to Both $f$-Divergence and Bregman Divergence Classes

Abstract

A divergence measure between two probability distributions or positive arrays (positive measures) is a useful tool for solving optimization problems in optimization, signal processing, machine learning, and statistical inference. The Csiszar f-divergence is a unique class of divergences having information monotonicity, from which the dual alpha geometrical structure with the Fisher metric is derived. The Bregman divergence is another class of divergences that gives a dually flat geometrical structure different from the alpha-structure in general. Csiszar gave an axiomatic characterization of divergences related to inference problems. The Kullback-Leibler divergence is proved to belong to both classes, and this is the only such one in the space of probability distributions. This paper proves that the alpha-divergences constitute a unique class belonging to both classes when the space of positive measures or positive arrays is considered. They are the canonical divergences derived from the dually flat geometrical structure of the space of positive measures.