Reducing data transfer in big-data workflows: the computation-flow delegated approach

Abstract

Existing orchestrated bioinformatics workflow execution approaches necessitate the transfer of datasets from biological data services to the analysis tool (computation) services of the workflow for various data analysis. This model of moving data to computation during workflow execution weakens the performance of the workflow especially when the orchestrated bioinformatics workflow has to handle big-data in it. Since the size of the analysis tools are much smaller than the datasets size in a workflow, in this paper, to minimize the dataflow and improve workflow performance, we propose a novel computation-flow delegated (CFD) approach. The CFD approach lets the tool services of the workflow to dynamically migrate analysis tools towards the datasets to perform computation on data side during workflow execution. We use a set of mobile agents to operate the CFD approach and present a mobile agent-based computation-flow delegation framework (MABCFD) to execute the workflow tasks. We implement the prototype of the MABCFD framework and analyze the performance of the CFD approach empirically by executing in isolation workflow patterns (sequence, fan-out and fan-in) common to bioinformatics applications. Performance analysis shows that the computation-driven CFD approach consistently outperforms the existing data-driven approaches across all patterns and scales favorably with data size.