Concerto: A Program Parallelization, Orchestration and Distribution Infrastructure

Abstract

The important step in Program Parallelization, is identifying the pieces of the given program, that can be run concurrently, on separate processing elements. The parallel pieces once identified, need to be hoisted and executed remotely, and the results combined. This is a complex process, usually referred to as Program Orchestration and Distribution, and the details are closely tied to the target architecture of the parallel machine. Program Distribution in a Shared Memory Parallel Computer, is comparatively simpler, and involves structuring the parallel pieces, as separate threads, with synchronization provided as needed, and just scheduling the threads, on the various processors. On the contrary, Program Orchestration on a Distributed Machine, such as a Cluster is more involved, and requires explicit message passing, with the help of Send and Receive primitives, to share variables between the parallel subprograms, which are running on separate machines. Concerto is a Parallelization, Orchestration and Distribution Framework, and is a component of our larger Program Transformation and Parallelization Solution. The parallel architectures targeted, include both Shared Memory Multicomputer and Distributed Memory Multicomputer. However, the focus of this paper, is mainly on the class of Distributed Memory Parallel Machines. Here we look at issues involved in Program Distribution, and provide a high level design of Concerto, our solution to the problem, along with the Program Parallelization and Distribution Algorithm. Majority of the existing Program Distribution Solutions, require user annotations to identify parallel pieces of code and data, which can be a cumbersome process, from the programmer perspective. However in Concerto, the Distribution and Mapping process is entirely automated, requires no user directives and is based solely on Dependence and Flow analysis of the sequential program.