Interoperable and network‐aware service workflows for big data executions at internet scale

Abstract

SummarySharing of computing resources and workload across different big data frameworks is challenging due to their lack of interoperable interfaces. In contrast, web services natively support an interoperable execution. Therefore, an increasing number of big data workflows are composed of data services and web service implementations that access and process big data. On the other hand, big data execution in the wide area networks needs to minimize latency and communication overheads to be able to scale seamlessly. Lack of network‐awareness of classic web service execution beyond data centers significantly challenges the scope of data services.Software‐Defined Networking (SDN) offers better control and management to the network, by unifying the control plane centrally, away from the distributed data plane devices. In this paper, we propose Software‐Defined Data Services (SDDS), an SDN‐based distributed service composition and workflow placement approach for data services in wide area networks. We first present the design of an SDDS framework that models the big data executions as composable data service workflows in multi‐domain network environments. We then evaluate the performance of a prototype SDDS framework through microbenchmarks. The benchmarks highlight the efficiency of SDDS in data service execution inside and beyond data centers.