Complexity Analysis of a Stochastic Variant of Generalized Alternating Direction Method of Multipliers

Abstract

Alternating direction method of multipliers (ADMM) receives much attention in the field of optimization and computer science, etc. The generalized ADMM (G-ADMM) proposed by Eckstein and Bertsekas incorporates an acceleration factor and is more efficient than the original ADMM. However, G-ADMM is not applicable in some models where the objective function value (or its gradient) is computationally costly or even impossible to compute. In this paper, we consider the two-block separable convex optimization problem with linear constraints, where only noisy estimations of the gradient of the objective function are accessible. Under this setting, we propose a stochastic linearized generalized ADMM (called SLG-ADMM) where two subproblems are approximated by some linearization strategies. By properly choosing algorithm parameters, we show, for objective function value gap and constraint violation, the worst-case $$\mathcal {O}\left( {1}/{\sqrt{k}}\right) $$ and $$\mathcal {O}\left( {\ln k}/{k}\right) $$ convergence rates in expectation measured by the iteration complexity for general convex and strongly convex problems respectively (k represents the iteration counter). For the latter case, we also obtain the convergence of the ergodic iterates generated by the proposed SLG-ADMM.