MIMD Machines Research Articles

A SIMD approach to parallel heuristic search

Serial search algorithms often exhibit exponential run times and may require an exponential amount of storage as well. Thus, the design of parallel search algorithms with limited memory is of obvious interest. This paper presents an efficient SIMD parallel algorithm, called IDPS (for iterative-deepening parallel search). At a broad level IDPS is a parallel version of IDA∗. While generically we have called our algorithm an IDPS, performance of four variants of it has been studied through experiments conducted on the well-known test-bed problem for search algorithms, namely the Fifteen Puzzle. During the experiments, data were gathered under two different static load balancing schemes. Under the first scheme, an unnormalized average efficiency of approximately 3 4 was obtained for 4K, 8K, and 16K processors. Under the second scheme, unnormalized average efficiencies of 0.92 and 0.76, and normalized average efficiencies of 0.70 and 0.63 were obtained for 8K and 16K processors, respectively. We show (as shown previously only for MIMD machines) that for admissible search, high average speedup can be obtained for problems of significant size. We believe that this research will enhance AI problem solving using parallel heuristic search algorithms.

Data management and control-flow aspects of an SIMD/SPMD parallel language/compiler

Features of an explicitly parallel programming language targeted for reconfigurable parallel processing systems, where the machine's N processing elements (PEs) are capable of operating in both the SIMD and SPMD modes of parallelism, are described. The SPMD (single program-multiple data) mode of parallelism is a subset of the MIMD mode where all processors execute the same program. By providing all aspects of the language with an SIMD mode version and an SPMD mode version that are syntactically and semantically equivalent, the language facilitates experimentation with and exploitation of hybrid SIMD/SPMD machines. Language constructs (and their implementations) for data management, data-dependent control-flow, and PE-address-dependent control-flow are presented. These constructs are based on experience gained from programming a parallel machine prototype and are being incorporated into a compiler under development. Much of the research presented is applicable to general SIMD machines and MIMD machines. >

Moving beyond message passing. Experiments with a distributed-data model

A message-passing model provides a natural and efficient parallel implementation for many applications in chemical physics on MIMD machines. However, although the distinction between local and non-local memory is at the very heart of writing efficient parallel programs, message passing leaves all responsibility for data management to the applications. This has significant, detrimental implications for both ease of programming and efficient use of shared and distributed resources. Examined here is a simple model which increments message passing with Linda-like tools for the manipulation of distributed-data structures. This is applied to common algorithms in chemical physics.

Parallel Processing for Supercomputing Speeds .

The various approaches to improving the speeds of computers by exploiting parallelism is briefly described. The architecture of PACE which is a loosely coupled, message-passing MIMD machine developed by ANURAG is described. PACE is an open architecture which is independent of the specific hardware used for the computing nodes. Several versions of PACE have been configured and the performance figures for some applications are presented.

Applying parallel computer systems to solve symmetric tridiagonal eigenvalue problems

A block parallel partitioning method for computing the eigenvalues of symmetric tridiagonal matrix is presented. The algorithm is based on partitioning, in a way that ensures load balance during computation. This method is applicable to both shared memory- and distributed memory-MIMD systems. Compared with other parallel tridiagonal eigenvalue algorithms existing in the literature, the proposed algorithm achieves a higher speedup of O( p) on a parallel computer with p-fold parallelism, which is linear, and the data communication between processors is less than that required for other methods. The results were tested and evaluated on an MIMD machine, and were within 62% to 98% of the predicted performance.

Processor interconnection networks from Cayley graphs

Cayley graphs of groups are presently being considered by the computer science community as models of architectures for large scale parallel processor computers. In the first section of this paper we discuss Cayley graphs and show how they may be used as a tool for the design and analysis of network architectures for these types of computers. Observing that routing on a Cayley graph is equivalent to a certain factoring problem in the associated group, we have been able to use a known powerful factoring technique in computational group theory to produce a fast efficient routing algorithm on the associated Cayley graph. In the second section of this paper we present this work. This research can be regarded as a first attempt to find general purpose routing algorithms for interconnection networks. Believing that average diameter of a network for a large scale MIMD machine is the predominant factor in determining network performance, we designed Cayley graphs to be used in a special study performed at the Supercomputing Research Center (SRC). The importance of the average diameter in determining network performance was supported by the fact that the graphs found by us had the smallest average diameter and outperformed all other graphs evaluated in the study. In fact, before being driven into saturation, one of our graphs sustained 9.4% more network traffic than the next best candidate, a butterfly architecture, and 74.3% better than the bench mark 2-d mesh. The last section of our paper is devoted to this work. This paper is divided into three sections. In the first section we discuss Cayley graphs and show how they may be used as a tool for the design and analysis of network architectures for parallel computers. In the second section we present our research on the routing problem. This research can be regarded as a first attempt to find general purpose routing algorithms for interconnection networks. In the last section we present some evidence that average diameter of a network for a large scale MIMD machine is the predominant factor in determining network performance.

Sparse iterative algorithm software for large‐scale MIMD machines: An initial discussion and implementation

AbstractThe parallelization of sophisticated applications has dramatically increased in recent years. As machine capabilities rise, greater emphasis on modeling complex phenomena can be expected. Many of these applications require the solution of large sparse matrix equations which approximate systems of partial differential equations (PDEs). Therefore we consider parallel iterative solvers for large sparse non‐symmetric systems and issues related to parallel sparse matrix software. We describe a collection of parallel iterative solvers which use a distributed sparse matrix format that facilitates the interface between specific applications and a variety of Krylov subspace techniques and multigrid methods. These methods have been used to solve a number of linear and non‐linear PDE problems on a 1024‐processor NCUBE 2 hypercube. Over 1 Gflop sustained computation rates are achieved with many of these solvers, demonstrating that high performance can be attained even when using sparse matrix data structures.

PASS—A multiuser parallel file system based on microcomputers

As computer applications have become more sophisticated, they have become data intensive. Such applications suffer from inadequate use of parallelism for processing data stored on secondary storage devices. Devices such as data base machines are useful in some applications, but many applications are too small or specialized to use data base machine technology. To bridge this gap, we have introduced the parallel secondary storage (PASS) system. PASS is based on a network of microcomputers. The individual microcomputers are assigned to a unit of secondary storage and the operations of the microcomputers are initiated and monitored by a control processor. The file system is capable of acting as either an SIMD or an MIMD machine depending, on the file type. Communication between the individual microcomputers and the control processor is described. The integration of the multiple microcomputers into the primitive operations on a file is examined. Finally, the strategies employed to enhance performance in the multiprogramming environment are discussed.

Parallel Computation of Geometry Control in Adaptive Truss Structures

The fast computation of geometry control in adaptive truss structures involves two distinct parts; namely, the efficient integration of the inverse kinematic dif ferential equations that govern the geometry control and the fast computation of the Jaco bian, which appears on the right-hand-side of the inverse kinematic equations. In this paper we present an efficient parallel implementation of the Jacobian computation on an MIMD machine. We obtain large speedup from the parallel implementation, which reduces the Jacobian computation to an O( M2/n ) procedure on an n-processor machine, where M is the number of members in the adaptive truss. The parallel algorithm given here is a good candidate for on-line geometry control of adaptive structures using attached processors.

Design and testing of a general-purpose neurocomputer

Here we describe the logical design and testing of a general-purpose neurocomputer, AMNIAC. It may be thought of as a programmable neural network that can simulate arbitrary SIMD and MIMD machines of practical interest (modeled as cellular automata, neural networks, or arbitrary automata networks). AMNIAC is purely bitwise (amnesic), i.e., requires no local memory or registers, other than short memory just long enough for a clock cycle. We discuss software serial and massively parallel simulations of AMNIAC as (a) tests of the logical design; and (b) benchmarks for evaluation of the trade-off between its universality and memory advantage versus overhead cost of mapping and speed. Theoretical applications of the design are given. A 3D SIMD version of AMNIAC which stabilizes if and only if its input network stabilizes (on the same input data) establishes the unsolvability of the stability problem for networks of finite bandwidth and also its weak solvability by an extension to a larger activation set. We also discuss the feasibility and trade-offs of a physical implementation.

Static scheduling for barrier MIMD architectures

In a SIMD or VLIW machine, conceptual synchronizations are accomplished by using a static code schedule that does not require run-time synchronization. The lack of run-time synchronization overhead makes these machines very effective for fine-grain parallelism, but they cannot execute parallel code structures as general as those executed by MIMD architectures, and this limits their utility.In this paper we present a timing analysis that allows a compiler for a MIMD machine to eliminate a large fraction of the run-time synchronization by making efficient use of static code scheduling. Although these techniques can be adapted to be applied to most MIMD machines, this paper centers on the analysis and scheduling for barrier MIMD machines. Barrier MIMDs are asynchronous multiple instruction stream/multiple data stream architectures capable of parallel execution of variable execution-time instructions and arbitrary control flow (e.g., while loops and calls). However, they also incorporate a special hardware barrier synchronization mechanism that facilitates static scheduling by providing a mechanism which the compiler can use to enforce precise timing constraints. In other words, the compiler tracks relative timing between processors and uses static code scheduling until the timing imprecision becomes too large, at which point the compiler simply inserts a barrier to reduce that timing imprecision to zero (or a small constant).This paper describes new scheduling and barrier placement algorithms for barrier MIMDs that are based loosely on the list scheduling approach employed for VLIWs [Ellis 1985]. In addition, the experimental results from scheduling thousands of synthetic benchmark programs for a parameterized barrier MIMD machine are presented.

A time-parallel multigrid-extrapolation method for parabolic partial differential equations

We consider the problem of solving unsteady partial differential equations on an MIMD machine. Conventional parallel methods use a data partitioning type approach in which the solution grid at each time-step is divided amongst the available processors. The sequential nature of the time integration is, however, retained. The algorithm presented in this paper makes use of a time-parallel approach, whreby several processors may be employed to solve at several time-steps simultaneously. The time-parallel method enables the inherent parallelism of the extrapolation scheme to be efficiently exploited, allowing a significant increase both in accuracy and in the degree of parallelism. The efficiencies obtained by an implementation on a message-passing multiprocessor demonstrate the suitability of the time-parallel extrapolation method for this type of equation.

Efficient Parallel Programming with Linda

Linda is a coordination language inverted by David Gelernter at Yale University, which when combined with a computation language (like C) yields a high-level parallel programming language for MIMD machines. Linda is based on a virtual shared associative memory containing objects called tuples. Skeptics have long claimed that Linda programs could not be efficient on distributed memory architectures. In this paper, we address this claim by discussing C-Linda's performance in solving a particular scientific computing problem, the shallow water equations, and make comparisons with alternatives available on various shared and distributed memory parallel machines.

On the parallel implementations of the linear Kalman and Lainiotis filters and their efficiency

In this paper, the parallel implementations of two well-known linear state-space filtering algorithms, namely the Kalman and the Lainiotis filters, in MIMD machines are studied from a computational standpoint. The analysis assumes both time invariant and time varying system models and uses precedence graphs and critical paths. The parallelism efficiency of the implementations is also defined and studied. Results indicate that these algorithms can be implemented in parallel using a comparatively small number of processors. Furthermore, the efficiency of the parallel implementations can be very high or very low, depending on the state and measurement vector dimensions. In dieser Arbeit werden parallele Realisierungen zweier bekannter Zustandsraum-Filteralgorithmen, nämlich der Kalman- und der Lainiotis-Filter, mit MIMD Prozessoren vom rechentechnischen Standpunkt aus untersucht. Die Analyse setzt sowohl zeitinvariante als auch zeitvariante Systemmodelle voraus und benutzt Präzedenzgraphen und kritische Pfade. Der Parallelitätswirkungsgrad der Realisierungen wird definiert und untersucht. Die Ergebnisse zeigen daβ diese Algorithmen parallel mit Hilfe einer vergleichsweise kleinen Anzahl von Prozessoren realisiert werden können. Weiterhin kann der Wirkungsgrad der Parallelrealisierungen sehr hoch oder sehr niedrig sein, abhängig von den Dimensionen des Zustands- und des Beobachtungsvektors. Nous étudions dans cet article, d'un point de vue calculatoire, les implantations paralléles sur des machines MIMD de deux algorithmes de filtrage linéaire dans l'espace d'état bien connus, à savoir les filtres de Kalman et de Lainiotis. Cette analyse est valable à la fois pour des modèles invariants et variants dans le temps, et utilise des graphes de priorité et des chemins critiques. L'efficacité des implantations en parallèle est également définie et étudiée. Les résultats indiquent que ces algorithmes peuvent être implantés en parrallèle à l'aide d'un nombre comparativement petit de processeurs. De plus, l'efficacité de ces implantations en parallèle peut être très grande ou très petite selon les dimensions des vecteurs d'état et de mesure.

Synchronization and load imbalance effects in distributed memory multi‐processor systems

AbstractSynchronization is a major cause of wasted computing cycles and of diminished performance in parallel computing. This paper investigates the effects of synchronization upon the performance of iterative methods on distributed memory MIMD machines. A quantitative analysis of the effects of the communication latency and of the load imbalance due to the non‐deterministic execution times for iterative methods is presented. This analysis explains the rather poor performance observed often in actual implementations of such methods and suggests better ways to achieve convergence without frequent synchronization.

Experiences with a parallel algorithm for data flow analysis

We have designed a family of parallel data flow analysis algorithms for execution on distributed-memory MIMD machines, based on general-purpose, hybrid algorithms for data flow analysis [Marlowe and Ryder 1990]. We exploit a natural partitioning of the hybrid algorithms and explore a static mapping, dynamic scheduling strategy. Alternative mapping-scheduling choices and refinements of the flow graph condensation used are discussed. Our parallel hybrid algorithm family is illustrated on Reaching Definitions, although parallel algorithms also exist for many interprocedural (e.g., Aliasing) and intraprocedural (e.g., Available Expressions) problems [Marlowe 1989]. We have implemented the parallel hybrid algorithm for Reaching Definitions on an Intel iPSC/2. Our empirical results suggest the practicality of parallel hybrid algorithms.

A parallel FFT algorithm for transputer networks

This paper presents a parallel FFT algorithm suitable for MIMD machines. It has the properties of uniform granularity, the dominance of computation, and suitability to large data matrices. The algorithm has been implemented for a tree and cubic types of transputer network in Occam language. A set of performance figures has been derived for different size of networks and initial matrices.

How to write parallel programs: a first course

In the not-too-distant future every programmer, software engineer, and computer scientist will need to understand parallelism, a powerful and proven way to run programs fast. The authors of this straightforward tutorial explain why this is so and provide the instruction that will transform ordinary programmers into parallel programmers.How to Write Parallel Programs focuses on programming techniques for the largest class of parallel machines - general purpose asynchronous or MIMD machines. It outlines the basic parallel algorithm classes and the three basic programming paradigms, takes up the implementation techniques for these paradigms, and presents a series of case studies explaining code and discussing its measured performance. Because parallel programming requires both a computing language and a coordination language, the authors use C and Linda (a language they developed) as a combination that can be simply and efficiently implemented on a wide range of machines. The techniques discussed, however, can be applied in any comparable language environment.Contents: Introduction. The Three Basic Models of Parallelism. Programming Techniques for the Three Basic Models. A Simple Problem, in Detail. Case Studies. From Parallelism to Coordination. Conclusions. Appendix: Linda User's Manual.

PARALLELIZATION OF SHORT- AND LONG-RANGE CELLULAR AUTOMATA ON SCALAR, VECTOR, SIMD AND MIMD MACHINES

Algorithms exhibiting parallelization on many different levels are discussed for short-and long-range cellular automata implemented on scalar, vector, SIMD and MIMD machines. Short range cellular automata are commonly used for simulating hydrodynamic fluid flows, while long range cellular automata are applicable to neural networks at zero temperature. A common programming approach based upon multi-spin coding and including higher levels of parallelization when possible, has been used to implement these models on the SUN SPARC-1, the IBM-3090, the Alliant FX/2800, the NEC-SX3/11, the Cray-YMP/832 and the Connection Machine, CM-2. Section 4 of the paper compares the performance of these computers for the algorithms discussed in the text. Additionally, the major subroutines for each computer type are given in the Appendix.

A communication mechanism and its implementation in the Multi-SIMD massively parallel machine SPHINX

This paper presents the communication mechanism for Multi-SIMD parallel computer SPHINX implemented as a set of MIMD machines. The described mechanism subdivides the controller of each SIMD machine in two levels, spliting thus the problems of accesses to the data and the management of communications. Communication mechanism includes synchronization functions. It uses floating channel objects implemented in the shared memory. The exclusive access to communication channels uses simple hardware mechanism based on a clock of the system, and does not require the multi-port memory.