The adventures of a simple algorithm

Abstract

This paper presents, analyzes, and tests a row-action method for solving the regularized linear programming problem: minimize 1/2∥ x ∥ 2 2+α c T x subject to A x ⩾ b , where α is a given positive constant. The proposed method is based on the observation that the dual of this problem has the form: maximize b T y −1/2∥A T y −α c ∥ 2 2 subject to y ⩾ 0 , and if y α solves the dual then the point x α=A T y α−α c provides the unique solution of the primal. Maximizing the dual objective function by changing one variable at a time results in an effective row-action method which is suitable for solving large sparse problems. One aim of this paper is to clarify the convergence properties of the proposed scheme. Let y k denote the estimated dual solution at the end of the kth iteration, and let x k=A T y k−α c denote the corresponding primal estimate. It is proved that the sequence { x k} converges to x α , while the sequence { y k} converges to a point y α that solves the dual. The only assumption which is needed in order to establish these claims is that the feasible region is not empty. Yet perhaps the more striking features of the algorithm are related to temporary situations in which it attempts to solve an inconsistent system. In such cases the sequence { y k} obeys the rule: y k= u k+k v , where { u k} is a fastly converging sequence and v is a fixed vector that satisfies A T v = 0 and b T v >0 . So the sequence { x k} is almost unchanged for many iterations. The paper ends with numerical experiments that illustrate the effects of this phenomenon and the close links with Kaczmarz's SOR method.