Joint and separate convexity of the bregman distance

Abstract

Algorithms involving Bregman projections for solving optimization problems have been receiving much attention lately. Several of these methods rely crucially on the joint convexity of the Bregman distance.In this note, we study joint and separate convexity of Bregman distances.To bring out the main ideas more clearly, we consider first functions defined on an open interval.Our main result states that the Bregman distance of a given function is jointly convex if and only if the reciprocal of its second derivative is concave. We observe that Bregman distances induced by the two most popular choices --- the \emph{energy} and the \emph{Boltzmann-Shannon entropy} --- are limiting cases in a profound sense.This result is generalized by weakening assumptions on differentiability and strict convexity.We then consider general, not necessarily separable, convex functions.The characterization of joint convexity has a natural and beautiful analog. Finally, we discuss spectral functions,where the situation is less clear.Throughout, we provide numerous examples to illustrate our results.