HEPart: A balanced hypergraph partitioning algorithm for big data applications

Abstract

Minimizing the query cost among multi-hosts is important to data processing for big data applications. Hypergraph is good at modelingdata and data relationships of complex networks, the typical big data applications, by representing multi-way relationships or interactions as hyperedges. Hypergraph partitioning (HP) helps to partition the query loads on several hosts, enabling the horizontal scaling of large-scale networks. Existing heuristic HP algorithms are generally vertex hypergraph partitioning, designed to minimize the number of cut hyperedges while satisfying the balance requirements of part weights regarding vertices. However, since workloads are mainly produced by group operations, minimizing query costs landing on hyperedges and balancing the workloads should be the objectives in horizontal scaling. We thus propose a heuristic hyperedge partitioning algorithm, HEPart. Specifically, HEPart directly partitions the hypergraph into K sub-hypergraphs with a minimum cutsize for vertices, while satisfying the balance constraint on hyperedge weights, based on the effective move of hyperedges. The performance of HEPart is evaluated using several complex network datasets modeled by undirected hypergraphs, under different cutsize metrics. The partitioning quality of HEPart is then compared with alternative hyperedge partitioners and vertex hypergraph partitioning algorithms. The experimental findings demonstrate the utility of HEPart (e.g. low cut cost while keeping load balancing as required, especially over scale-free networks).