A Relaxed Synchronization Approach for Solving Parallel Quadratic Programming Problems with Guaranteed Convergence

Abstract

In this paper we present a novel numerical algorithm for efficiently solving large-scale quadratic programming problems in massively parallel computing systems. The main challenge in maximizing processor utilization is to reduce idling due to synchronization across processors. Typically, synchronization is necessary after every iteration, which prevents many numerical algorithms from scaling with number of processors. We relax this requirement by synchronizing at a lower rate, which is referred to as lazy synchronization. We show analytically and experimentally that lazy synchronization is numerically stable and converges to the same result as the conventional tightly synchronized implementation. Furthermore, the convergence speed of the proposed algorithm is faster with lazy synchronization. The numerical stability, convergence rate and the optimal rate for synchronization are analytically shown. The proposed algorithm is implemented in a 40-node distributed system in the Amazon Elastic Computing infrastructure. We show a 160 times speedup in solution time for a large-scale quadratic programming problem using a synthetic dataset. The experiments demonstrate that the use of relaxed synchronization technique reduces communication overhead in the distributed systems by 99.65% in comparison to the tightly synchronization implementation.