Optimization and scheduling algorithm for data intensive workflows in distributed data mining architecture

Abstract

New Grid and cloud solutions for distributed data mining and data processing are needed for execution of data intensive workflows. In contrast of the standard workflows, in which data between the jobs are exchanged in the form of files and the jobs are finished when they process the input data, data intensive workflows receive data organized in blocks which are streamed on inputs, analyze the data and produce stream output. Each job is active for a long period of time and can receive new data. In our previous research works we proposed the Open Grid Service Architecture for Data Mining (OGSA-DM), which is capable of executing data intensive workflows. According to our analysis, the current algorithms for scheduling workflows can't be applied on data intensive workflows because they produce unsatisfactory results and can't guarantee optimal solution. In this paper we propose new optimization and scheduling algorithm which is developed on the advantages of data intensive workflows. In several experiments we've shown that our proposed algorithm works and gives satisfactory results.