Towards Better Shared Memory Programming Models

Abstract

In an often-cited paper, A. Karp [1] commented on the “sorry state” of parallel programming, lamenting the difficulties in programming and debugging parallel programs on existing programming models. Indeed, a variety of programming models have been proposed with not entirely satisfactory results. In this note, we focus on the world of massively parallel shared memory machines and examine programming models for these machines. A shared memory parallel computer is one in which the processors communicate by reading and writing data words to a shared memory. We consider shared memory parallel computers with the following characteristics. Each processor has some fast, relatively small, private memory (e.g. registers, cache). There is a large shared memory composed of a series of memory banks that are accessible by all processors via an interconnection network. Each processor has one such memory bank which is local to it, i.e. it can access the bank without going through the interconnection network. The access time to a memory bank for a processor depends in part on the lengths of the links traversed and the number of hops in the path from the processor to the bank. Special coprocessors take care of interprocessor communication: both programmers and processors are relieved of the burden of routing the parallel requests to shared memory, including handling intermediate hops of requests traveling to their destinations. In order to overcome the high latency of accessing global memory, the parallel computer supports the pipelining of global memory accesses through the network. This implies that the interconnection network has sufficient bandwidth to handle the multiple requests per processor.