Online Balanced Repartitioning

Abstract

Distributed cloud applications, including batch processing, streaming, and scale-out databases, generate a significant amount of network traffic and a considerable fraction of their runtime is due to network activity. This paper initiates the study of deterministic algorithms for collocating frequently communicating nodes in a distributed networked systems in an online fashion. In particular, we introduce the Balanced RePartitioning (BRP) problem: Given an arbitrary sequence of pairwise communication requests between n nodes, with patterns that may change over time, the objective is to dynamically partition the nodes into $$\ell $$ clusters, each of size k, at a minimum cost. Every communication request needs to be served: if the communicating nodes are located in the same cluster, the request is served locally, at cost 0; if the nodes are located in different clusters, the request is served remotely using inter-cluster communication, at cost 1. The partitioning can be updated dynamically (i.e., repartitioned), by migrating nodes between clusters at cost $$\alpha $$ per node migration. The goal is to devise online algorithms which find a good trade-off between the communication and the migration cost, i.e., “rent” or “buy”, while maintaining partitions which minimize the number of inter-cluster communications. BRP features interesting connections to other well-known online problems. In particular, we show that scenarios with $$\ell =2$$ generalize online paging, and scenarios with $$k=2$$ constitute a novel online version of maximum matching. We consider settings both with and without cluster-size augmentation. Somewhat surprisingly (and unlike online paging), we prove that any deterministic online algorithm has a competitive ratio of at least k, even with augmentation. Our main technical contribution is an $$O(k \log {k})$$ -competitive deterministic algorithm for the setting with (constant) augmentation. This is attractive as, in contrast to $$\ell $$ , k is likely to be small in practice. For the case of matching ( $$k=2$$ ), we present a constant competitive algorithm that does not rely on augmentation.