A unified modeling framework for distributed resource allocation of general fork and join processing networks

Abstract

This paper addresses the problem of distributed resource allocation in general fork and join processing networks. The problem is motivated by the complicated processing requirements arising from distributed data intensive computing. In such applications, the underlying data processing software consists of a rich set of semantics that include synchronous and asynchronous data fork and data join. The different types of semantics and processing requirements introduce complex interdependence between various data flows within the network. We study the distributed resource allocation problem in such systems with the goal of achieving the maximum total utility of output streams. Past research has dealt with networks with specific types of fork/join semantics, but none of them included all four types. We propose a novel modeling framework that can represent all combinations of fork and join semantics, and formulate the resource allocation problem as a convex optimization problem on this model. We propose a shadow-queue based decentralized iterative algorithm to solve the resource allocation problem. We show that the algorithm guarantees optimality and demonstrate through simulation that it can adapt quickly to dynamically changing environments.