Communication-Minimal Partitioning and Data Alignment for Affine Nested Loops

Abstract

Data alignment and computation domain partitioning techniques have been widely investigated to reduce communication overheads in distributed memory systems. This paper considers how these two techniques can be combined and applied to affine nested loops. Current data alignment techniques focus on individual entries of data arrays and, in general, cannot be used directly for cases when blocks of entries should be aligned collectively. This paper shows that existing data alignment techniques can be applied to partitioned algorithms if the null space of the data array indexing matrix is a boundary of computation blocks or the intersection of some computation block boundaries. These conditions can be used to generate several different partitionings and time-space transformations for a given target architecture. An example illustrates how it is possible to trade-off the number of communications and the memory space. Another example shows partitions of matrix-matrix multiplication that have smaller communication-computation ratios than Cannon's algorithm.