Improved Power of Two Choices for Fat-Tree Routing

Abstract

The fat-tree networking topology have gained prominence in various parallel and distributed systems such as high-performance computing clusters and data centers. To support high throughput and low latency applications, effective load-balancing schemes are in great demand. However, the commonly deployed scheme, equal-cost multipath, suffers from severe hash collisions that lead to poor performance. Recently, another realization of randomized load balancing, DRILL, has been proposed, which adopts the two-choice algorithm to achieve in-network local-to-switch load balancing. Although DRILL can well balance uplink traffic, it shows some limitations on alleviating downlink contentions. Motivated by this observation, we propose a thresholded two-choice (TTC) scheme, which modifies the two-choice algorithm such that it can balance both uplink and downlink traffic. To balance downlink traffic, TTC sets a default path for every source-destination pair using the D-mod-k scheme. The rationale for this is that D-mod-k minimizes the level of path collision over downlinks for any permutation in a fat-tree network. To balance uplink traffic, TTC makes dynamic path decisions using an algorithm that is based on the two-choice idea. To better leverage default paths in downlink load balancing, it is desirable that TTC routes the majority of traffic onto the default paths. To this end, we introduce a new thresholding mechanism to the two-choice algorithm, which contributes to a better overall performance. Our analysis shows that the introduced thresholding mechanism does not significantly affect the uplink load balancing performance. Moreover, our numerical study indicates that TTC can achieve a better system-wide performance than DRILL.