Optical Interconnectivity in a Scalable Data-Parallel System

Abstract

Scalability. Few would disagree that the future of high-performance computing lies with massively parallel systems(MPS), since there are major physical limitations to the clockrate of a single processor. Massively parallel systems arerequired to be scalable in the sense that their performanceshould be proportional to the number of processors. However,a feasible architecture for a scalable massively parallel systemis still wanting, as true, i.e., unlimited scalability is not onlytheoretically impossible but even in the practical sense cannotbe achieved on a range of more than an order of magnitude inthe number of processors. Whatever a system’s architecture,interconnect, or programming model, something will not scale:the throughput or latency of the interconnect, its cost, orthe synchronization overheads. Since all these componentscontribute to performance in different ways, the issue of scal-ability is a very complex one indeed.So in what sense can one argue for scalability of massivelyparallel systems? There have been quite a few attempts todefine it (see [1]) on the basis of some strong assumptionsregarding the nature of parallel computation. The mostcommon assumption that is being made in such analyses is thatthe processors run some predominantly local processes whichrequire little external communication (it manifests itself in theparameter “communication to computation ratio” assumed tobe small and used in all but a few performance models). Bymigrating these self-contained processes and placing severalof them per processor to balance out computational workacross the system, it is believed that scalability may beachieved without requiring physically unfeasible networkingand/or dramatically different computational models. In reality,however, process concurrency of such kind continues to facevarious fundamental limitations from data mapping to dynamicload balancing to program paradigm issues.Data parallelism. We would like to address a simpler, andin our view more promising, way of doing parallel computing