Coded Distributed Computing: Straggling Servers and Multistage Dataflows

Abstract

In this paper, we first review the Coded Distributed Computing (CDC) framework, recently proposed to significantly slash the data shuffling load of distributed computing via coding, and then discuss the extension of the CDC techniques to cope with two major challenges in general distributed computing problems, namely the straggling servers and multistage computations. When faced with straggling servers in a distributed computing cluster, we describe a unified coding scheme that superimposes CDC with the Maximum-Distance-Separable (MDS) coding on computation tasks, which allows a flexible tradeoff between computation latency and communication load. On the other hand, for a general multistage computation task expressed as a directed acyclic graph (DAG), we propose a coded framework that given the load of computation on each vertex of the DAG, applies the generalized CDC scheme individually on each vertex to minimize the communication load.