Convergence of alternating optimization

Abstract

Let f : Rs → R be a real-valued function, and let x = (x1,...,xs)T ∈ Rs be partitioned into t subsets of non-overlapping variables as x = (X1,...,Xt)T, with Xi ∈ Rpi for i = 1,...,t, Σi=1tpi = s. Alternating optimization (AO) is an iterative procedure for minimizing f(x) = f(X1, X2,..., Xt) jointly over all variables by alternating restricted minimizations over the individual subsets of variables X1,...., Xt. Alternating optimization has been (more or less) studied and used in a wide variety of areas. Here a self-contained and general convergence theory is presented that is applicable to all partitionings of x. Under reasonable assumptions, the general AO approach is shown to be locally, q-linearly convergent, and to also exhibit a type of global convergence.