Abstract
In this work we show that randomized (block) coordinate descent methods can be accelerated by parallelization when applied to the problem of minimizing the sum of a partially separable smooth convex function and a simple separable convex function. The theoretical speedup, as compared to the serial method, and referring to the number of iterations needed to approximately solve the problem with high probability, is a simple expression depending on the number of parallel processors and a natural and easily computable measure of separability of the smooth component of the objective function. In the worst case, when no degree of separability is present, there may be no speedup; in the best case, when the problem is separable, the speedup is equal to the number of processors. Our analysis also works in the mode when the number of blocks being updated at each iteration is random, which allows for modeling situations with busy or unreliable processors. We show that our algorithm is able to solve a LASSO problem involving a matrix with 20 billion nonzeros in 2 h on a large memory node with 24 cores.
Highlights
1.1 Big data optimizationRecently there has been a surge in interest in the design of algorithms suitable for solving convex optimization problems with a huge number of variables [12,15]
The number of iterations a Coordinate descent methodsCoordinate descent methods (CDM) requires to solve a smooth convex optimization problem is O( nL R2 ), where is the error tolerance, n is the number variables, Lis the average of the Lipschitz constants of the gradient of the objective function associated with the variables and R is the distance from the starting iterate to the set of optimal solutions
Complexity We show theoretically (Sect. 7) and numerically (Sect. 8) that parallel coordinate descent methods (PCDMs) accelerates on its serial counterpart for partially separable problems
Summary
There has been a surge in interest in the design of algorithms suitable for solving convex optimization problems with a huge number of variables [12,15]. The size of problems arising in fields such as machine learning [1], network analysis [29], PDEs [27], truss topology design [16] and compressed sensing [5] usually grows with our capacity to solve them, and is projected to grow dramatically in the decade. Much of computational science is currently facing the “big data” challenge, and this work is aimed at developing optimization algorithms suitable for the task
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
