On communication efficient dataflow computing in software defined networking enabled cloud

Abstract

SummaryDataflow computing has become a promising computing paradigm as an alternative to traditional control‐centric computing paradigm to facilitate big data processing. Big data process often happens in cloud computing environment as the datacenter provisions a large amount of resource. Dataflow computing, as a data‐centric computing paradigm, requires the dataflows to be shuffled among different codelets (ie, data processing units) deployed in the datacenter servers. It is significant to well schedule the dataflow transferring for communication efficiency. It is highly regarded that the datacenter network shall be managed by software defined networking (SDN) technology for flexibility consideration. In SDN managed datacenter, a dataflow requires a forwarding rule in the forwarding table of each switch on its routing path. However, the SDN switches are limited in the forwarding table size. This introduces an unignorable issue in the codelet deployment problem. Therefore, we are motivated to take such forwarding table size constraints into the problem of dataflow codelet deployment in the datacenters managed by SDN. In particular, we aim at minimizing the communication cost efficiency while guarantee the dataflow computing performance at the same time. The communication cost minimization problem is formulated into an integer linear programming form, which is relaxed to design a heuristic algorithm. The experiment results show that our relaxation algorithm can significantly improve the communication cost efficiency via ingenious codelet placement.