Speeding up distributed join computation

Abstract

We study join computation in the presence of join-attribute skew on distributed architectures, in particular clusters of commodity machines. Our goal is to determine how to partition data and computation, in order to minimize running time. Specifically, our focus is not only on equi-joins, but also band-joins, which generalize equi-joins. For both types of join queries, there exists a tradeoff between load balance and data duplication. Skew can be addressed by more fine-grained partitioning, at the cost of input duplication. Different partitionings lead to various tradeoffs between load balance and data duplication. To avoid exhaustive exploration of the partitioning-parameter space, we propose methods that intelligently search for a partitioning that achieves the sweet spot in this tradeoff with the help of a novel running-time model. This model can also be directly integrated into the running-time optimization process, reducing optimization cost by orders of magnitude.