Cascaded Coded Distributed Computing Schemes Based on Symmetric Designs

Abstract

Coded distributed computing (CDC) is an efficient method to reduce the communication load in general distributed computing frameworks such as MapReduce. In these systems, one usually needs to split the data set into disjoint files and design several output functions to complete a computational task. Li et al. provided some CDC schemes achieving optimal communication load. However, as the number of computing nodes increases, the numbers of input files and output functions of such schemes grow too fast to be applied in practice. In this paper, several infinite families of cascaded CDC schemes, where each output function is computed multiple times, are constructed by taking advantage of symmetric designs. Most importantly, the numbers of input files and output functions of each new scheme are linear with the number of computing nodes and the communication load of each new scheme approximates to that of the CDC scheme derived by Li et al. when the number of nodes becomes large.