Network Resource Control for Data Intensive Applications in Heterogeneous Infrastructures

Abstract

Many scientific workflow applications are data intensive. Large quantities of simulation generated data or data collected from sensors or instruments drive the process flow of the applications, and the processing of the data is commonly done at a different location from where the data is stored. Advanced networks such as hybrid networks make it feasible for high level applications to request network paths and service provisioning. However, current applications tune the execution quality rarely considering network resources, and by selecting only optimal software services and computing resources. Including network services in the resource scheduling adds an extra dimension workflow applications to optimize the runtime performance but also introduces new challenges, in particular in heterogeneous infrastructures. In this paper we continue our previous work on a system called Network QoS Planner (NEWQoSPlanner) which is able to select network resources in the context of workflow composition and scheduling. We will discuss how NEWQoSPlanner invokes network services to achieve dynamic resource optimization for workflows, and how to apply such planner in heterogeneous infrastructures.