A framework for scheduling and managing big data applications in a distributed infrastructure

Abstract

Nowadays, big data has received attention from researchers, business industries, education, and scientific communities. Big data analytics has to deal with large scale data that consist of both structured and unstructured data. These data are to be handled properly, that is extracting, processing, and analyzing those data to obtain meaningful information from them in a limited time. To yield insightful information, the processing of big data analytics requires high performance computing system, storage, and network resources. Hence, it is essential to design a high performance computing infrastructure with sufficient bandwidth which is capable to handle the big data processing in an efficient manner. However, the current network architectures in those infrastructures, with predefined network policies, do not allow for just-in-time reconfiguration of the networking infrastructure as demanded by big data analytics. In addressing these limitations, Software-Defined Networking (SDN) offers the means to dynamically configure the network parameters, dynamically provision the networks, and the network itself can be sliced in an on-demand manner. This research aims to characterize SDN with respect to the demands of big data analytics in Cluster, Grid, and Cloud Computing resources. The main motivation behind this research study is to design and develop an intelligent framework named as Big Data Analytics Management System (BDAMS) for collectively managing the compute, storage, and network resources in Cluster, Grid, and Cloud infrastructure for big data analytics.