Performance Analysis of Loosely Coupled Applications in Heterogeneous Distributed Computing Systems

Abstract

Loosely coupled applications composed of a potentially very large number (from tens of thousands to even billions) of tasks are commonly used in High-Throughput Computing (HTC) and Many-Task Computing (MTC) paradigms. To efficiently execute large-scale computations which can exceed the capability in a single type of computing resources within expected time, we should be able to effectively integrate resources from Heterogeneous Distributed Computing (HDC) systems such as Clusters, Grids, and Clouds. In this paper, we quantitatively analyze the performance of three different real scientific applications consisting of many tasks on top of HDC systems based on a Partnership of Distributed Computing Clusters, Grids, and Clouds to show practical issues that normal scientific users can face during the course of leveraging these systems. Our experimental results show that the performance of a loosely coupled application can be significantly affected by the characteristics of a HDC system, along with hardware specification of a node, and their impacts on the performance can vary widely depending on the resource usage pattern of each application. Throughout our extensive performance study with representative HDC systems and real scientific applications, we aim to give an insight to the research community on design and implementation of a next generation middleware system that can intelligently support large-scale loosely coupled applications by considering both of resource and application perspectives.