Single-instruction Multiple-data Machine Research Articles

Divergence analysis

Growing interest in graphics processing units has brought renewed attention to the Single Instruction Multiple Data (SIMD) execution model. SIMD machines give application developers tremendous computational power; however, programming them is still challenging. In particular, developers must deal with memory and control-flow divergences. These phenomena stem from a condition that we call data divergence, which occurs whenever two processing elements (PEs) see the same variable name holding different values. This article introduces divergence analysis, a static analysis that discovers data divergences. This analysis, currently deployed in an industrial quality compiler, is useful in several ways: it improves the translation of SIMD code to non-SIMD CPUs, it helps developers to manually improve their SIMD applications, and it also guides the automatic optimization of SIMD programs. We demonstrate this last point by introducing the notion of a divergence-aware register spiller. This spiller uses information from our analysis to either rematerialize or share common data between PEs. As a testimony of its effectiveness, we have tested it on a suite of 395 CUDA kernels from well-known benchmarks. The divergence-aware spiller produces GPU code that is 26.21% faster than the code produced by the register allocator used in the baseline compiler.

A Coarse-Grain Hierarchical Technique for 2-Dimensional FFT on Configurable Parallel Computers

FPGAs (Field-Programmable Gate Arrays) have been widely used as coprocessors to boost the performance of data-intensive applications [1], [2]. However, there are several challenges to further boost FPGA performance: the communication overhead between the host workstation and the FPGAs can be substantial; large-scale applications cannot fit in a single FPGA because of its limited capacity; mapping an application algorithm to FPGAs still remains a daunting job in configurable system design. To circumvent these problems, we propose in this paper the FPGA-based Hierarchical-SIMD (H-SIMD) machine with its codesign of the Pyramidal Instruction Set Architecture (PISA). PISA comprises high-level instructions implemented as FPGA functions of coarse-grain SIMD (Single-Instruction, Multiple-Data) tasks to facilitate ease of program development, code portability across different H-SIMD implementations and high performance. We assume a multi-FPGA board where each FPGA is configured as a separate SIMD machine. Multiple FPGA chips can work in unison at a higher SIMD level, if needed, controlled by the host. Additionally, by using a memory switching scheme and the high-level PISA to partition applications into coarse-grain tasks, host-FPGA communication overheads can be hidden. We enlist the two-dimensional Fast Fourier Transform (2D FFT) to test the effectiveness of H-SIMD. The test results show sustained high performance for this problem. The H-SIMD machine even outperforms a Xeon processor for this problem.

Development of a parallel molecular dynamics code on SIMD computers: Algorithm for use of pair list criterion

In recent years several implementations of molecular dynamics (MD) codes have been reported on multiple instruction multiple data (MIMD) machines. However, very few implementations of MD codes on single instruction multiple data (SIMD) machines have been reported. The difficulty in using pair lists of nonbonded interactions is the major problem with MD codes for SIMD machines, such that, generally, the full connectivity computation has been used. We present an algorithm, the global cut-off algorithm (GCA), which permits the use of pair lists on SIMD machines. GCA is based on a probabilistic approach and requires the cut-off condition to be simultaneously verified on all nodes of the machine. The MD code used was taken from the GROMOS package; only the routines involved in the pair lists and in the computation of nonbonded interactions were rewritten for a parallel architecture. The remaining calculations were performed on the host computer. The algorithm has been tested on Quadrics computers for configurations of 32, 128, and 512 processors and for systems of 4000, 8000, 15,000, and 30,000 particles. Quadrics was developed by Istituto Nazionale di Fisica Nucleare (INFN) and marketed by Alenia Spazio. © 1998 John Wiley & Sons, Inc. J Comput Chem 19: 685–694, 1998

Cellular interconnects optimization algorithm for optoelectronic single-instruction multiple data

We present a novel algorithm for designing optimal cellular interconnects (OCI's), which can significantly accelerate the communications among processors in single-instruction multiple-data machines with optoelectronic interconnections. We present the foundations of the OCI architecture and show that the optoelectronic OCI is the optimal topology for a space-invariant interconnect pattern. The OCI is optimal in achieving a minimum number of clock cycles per data shift for a given number of optoelectronic links. In addition, our algorithm for designing the OCI is deterministic, whereas previous designs required a trial-and-error procedure.

Tree decomposition strategy for massively parallel game tree searching

A new tree decomposition strategy, EYBWC (enhanced young brothers wait concept), is proposed for two-player control game tree searching on massively parallel machines. Based on the YBWC (young brothers wait concept) which is one of the best existing tree decomposition strategies, this new strategy notably alleviates processor starvation in a massively parallel search by providing many more nodes for processors to search in parallel. An experiment on the Maspar MP-2 SIMD (single instruction multiple data) machine with 1024 processors shows that EYBWC is far more efficient than YBWC.

Iterative optimization of diffractive phase elements simultaneously implementing several optical functions

The design of diffractive optical elements that incorporate several optical functions in a single element is discussed. The technique used involves iterative optimization. Aprevious paper is continued, in which initial results with few sampling points were reported. Here new results that involve a large number of sampling points are reported. Because the algorithm is computationally intensive with a large number of data points, the parallel implementation of the algorithm on a MASPAR machine is described. MASPAR is a single-instruction multiple-data machine with 16,384 processors. The computer simulations discussed involve simultaneous wavelength demultiplexing, focusing, and the filtering out of a particular wavelength component. It is shown that satisfactory designs of diffractive optical elements can be achieved by the assignment of only a small number of sampling points on the output plane that adequately specify what is required at each wavelength.

Parallel I/O subsystems in massively parallel supercomputers

Applications on MPPs often require a high aggregate bandwidth of low-latency to secondary storage. This requirement can met by internal parallel subsystems that comprise dedicated nodes, each with processor, memory, and disks.Massively parallel processors (MPPs), encompassing from tens to thousands of processors, are emerging as a major architecture for high-performance computers. Most major computer vendors offer computers with some degree of parallelism, and many smaller vendors specialize in producing MPPs. These machines are targeted for both grand-challenge problems and general-purpose computing.Like any computer, MPP architectural design must balance computation, memory bandwidth and capacity, communication capabilities, and I/O. In the past, most design research focused on the basic compute and communications hardware and software. This led to unbalanced computers that had relatively poor performance. Recently, researchers have focused on designing hardware and software for subsystems in MPPs. Consequently, most current MPPs have an architecture based on an internal parallel subsystem (the Architectures with parallel I/O sidebar describes some examples). In these computers, this subsystem encompasses a collection of nodes, each managing and providing access to a set of disks. The nodes connect to other nodes in the system by the same switching network that connects the compute nodes.In this article we'll examine why many MPPs use parallel subsystems, what architecture is best for such a subsystem, and how to implement the subsystem. We'll also discuss how parallel file systems and their user interfaces can exploit the parallel to provide enhanced services to applications.The systems discussed in this article are mostly tightly coupled distributed-memory MIMD (multiple-instruction, multiple-data) MPPs. In some cases, we also discuss shared-memory and SIMD (single-instruction, multiple-data) machines. We'll discuss three node types. Compute nodes are optimized to perform floating-point and numeric calculations, and have no local disk except perhaps for paging, booting, and operating-system software. nodes contain the system's secondary storage, and provide the parallel file-system services. Gateway nodes provide connectivity to external data servers and mass-storage systems. In some cases, individual nodes can serve as more than one type. For example, the same nodes often handle and gateway functions. The Terminology sidebar defines some other terms used in this article.

High-speed free-space interconnect based on optical ring topology: experimental demonstration

The design and implementation of a high-speed optical-ring-topology-based free-space optical interconnect is described. This interconnect system operates at 500 MHz and consists of 16 laser transmitters, a four-channel free-space interconnect, and a high-speed receiver. A nearest-neighbor interconnect is demonstrated. At the data rate of 500 MHz the total system throughput is 8 Gbits/s. The system can easily be operated at much higher data rates since the rate is limited only by the electronic circuitry. A discussion is given about device issues such as optical switching devices, and practical system-design issues such as integration and interface with current electronic systems are considered. This interconnect is promising to the implementation of ultrafast massively parallel single-instruction multiple-data machines.

Distribution algorithms for document allocation in multiprocessor information retrieval systems

Distributed memory information retrieval systems have been used as a means of managing the vast volume of documents in an information retrieval system, and to improve query response time. However, proper allocation of documents plays an important role in improving the performance of such systems. Maximising the amount of parallelism can be achieved by distributing the documents, while the inter-node communication cost is minimised by avoiding documents distribution. Unfortunately, these two factors contradict each other. Finding an optimal allocation satisfying the above objectives is referred to as distributed memory document allocation problem (DDAP), and it is an NP-Complete problem. Heuristic algorithms are usually employed to find an optimal solution to this problem. Genetic algorithm is one such algorithms. In this paper, a genetic algorithm is developed to find an optimal document allocation for DDAP. Several well-known network topologies are investigated to evaluate the performance of the algorithm. The approach relies on the fact that documents of an information retrieval system are clustered by some arbitrary method. The advantages of a clustered document approach specially in a distributed memory information retrieval system are well-known. Since genetic algorithms work with a set of candidate solutions, parallelisation based on a Single Instruction Multiple Data (SIMD) paradigm seems to be the natural way to obtain a speedup. Using this approach, the population of strings is distributed among the processing elements. Each string is processed independently. The performance gain comes from the parallel execution of the strings, and hence, it is heavily dependent on the population size. The approach is favoured for genetic algorithms' applications where the parameter set for a particular run is well-known in advance, and where such applications require a big population size to solve the problem. DDAP fits nicely into the above requirements. The aim of the parallelisation is two-fold: the first one is to speedup the allocation process in DDAP which usually consists of thousands of documents and has to use a big population size, and second, it can be seen as an attempt to port the genetic algorithm's processes into SIMD machines.

Computer simulation of materials using parallel architectures

Algorithms are designed to implement molecular dynamics (MD) and quantum molecular dynamics (QMD) simulations on emerging concurrent architectures. For systems with finite-range interactions, a domain-decomposition algorithm is used to implement the multiple-time-step (MTS) approach to MD simulations on distributed-memory multiple instruction multiple data (MIMD) machines. Parallel algorithms are also designed for MD simulations of bulk Coulombic systems. The performances of these algorithms are tested on the Intel iPSC/860 system. The computational complexities of these algorithms are O( N) and parallel efficiencies close to 0.9. The core computational kernel of the QMD approach consists of solutions of parabolic partial differential equations (PDE) such as the time-dependent Schrödinger equation or time-dependent Kohn-Sham equation. This problem is coupled with another computationally intensive problem, i.e., solution of elliptic PDEs (the Poisson equation) for the long-range electron-electron interaction. We have designed parallel algorithms for both problems on SIMD (single instruction multiple data) machines. In the past two years, we have used the parallel computer architectures in our Concurrent Computing Laboratory for Materials Simulations (CCLMS) to carry out MD and QMD simulations on network glasses, ceramic composites, nanophase materials, solid C 60 and graphitic tubules, and quantum transport in nanoscale devices. Structural transformation, intermediate-range order and dynamic behavior of SiO 2 glass at high pressures are investigated with molecular dynamics. At high densities, the height of the first sharp diffraction peak is considerably diminished, its position changes from 1.6 to 2.2 Å −1, and a new peak appears at 2.85 Å -1. At twice the normal density, SiO bond length increases, SiO coordination changes from 4 to 6 and OSiO bond-angle changes from 109° to 90°. This is a tetrahedral to octahedral transformation, which was reported recently by Meade, Hem1ey, and Mao. Molecular dynamics simulations of porous silica, in the density range 2.2-0.1 g/cm 3, are carried out on a 41 472 particle system using a MIMD computer. The internal surface area, pore surface-to-volume ratio, pore-size distribution, fractal dimension, correlation length and mean-particle size are determined as a function of the density. Structural transition between a condensed amorphous phase and a low-density porous phase is characterized by these quantities. Various dissimilar porous structures with different fractal dimensions are obtained by controlling the preparation schedule and temperature.

The GF11 parallel computer

GF11 is a parallel computer operational at IBM's T.J. Watson Research Center. It is based on the SIMD (Single Instruction Multiple Data) model of parallel computing. GF11 attains its peak execution rate of 11.3 GigaFlops by using 566 identical processing elements, each capable of delivering 20 MegaFlops. Each processor has its own 64 Kb static RAM that can access a 32-bit word on each floating point operation, a 2 Mb dynamic RAM that operates at one fourth of the SRAM speed, and a 1 Kb register file that provides four accesses per floating point operation. The processors communicate through a 576×576 Benes network, organized as three stages of 24×24 crossbar switches. The network provides 11.3 Gb/sec of communication bandwidth to the processors and allows the processors to dynamically reconfigure themselves into arrays f various dimensions and sizes or other interesting interconnection patterns such as a tree, hypercube, etc. This configuration can take place on every word transfer without sacrificing the bandwidth. GF11 has several architectural enhancements to circumvent the limitations of the standard SIMD model such as the ability to perform multiple operations in every instruction and the ability to modify the operations occurring within individual processors based on processor specific data. Preliminary benchmarking efforts on some applications indicate that near peak performance can be sustained on most applications, including some that were previously believed to be ill suited SIMD machines. Minimal restructuring of programs and algorithms is required for achieving this performance. The architecture of GF11 is summarized in this paper and the implementations of Finite Element analysis, LU decomposition, Gaussian Elimination, and Fast Fourier Transform are discussed to illustrate GF11's ability to deliver good performance with minor program restructing.

Depth-first heuristic search on a SIMD machine

We present a parallel implementation of Iterative-Deepening-A∗, a depth-first heuristic search, on the single-instruction, multiple-data (SIMD) Connection Machine ★ ★ Connection Machine is a trademark of Thinking Machines Corporation. . Heuristic search of an irregular tree represents a new application of SIMD machines. The main technical challenge is load balancing, and we explore three different techniques in combination. We also use a simple method for dynamically determining when to stop searching and start load balancing. We achieve an efficiency of 69%, for a speedup of 5685 on 8K processors, an efficiency of 64%, for a speedup of 10,435 on 16K processors, and an efficiency of 53%, for a speedup of 17,300 on 32K processors on the Fifteen Puzzle. On hard problem instances, we achieved efficiencies as high as 80%, for a speedup of 26,215 on 32K processors. Our analysis indicates that work only needs to increase as P log P to maintain constant efficiency, where P is the number of processors. This high degree of scalability was confirmed empirically for the range of 16 to 32,768 (32K) processors.

Air quality simulation through cellular automata

In this paper we present some applications of Cellular Automata to environmental problems and a model for air pollution simulation based on this approach. The proposed model differs from those based on the analytical or numerical solution of the transport and diffusion equation because there is no attemp to mimic continuous variables, but the discretization in space and time is intrinsic in the model. The model, though based on a very simple, linear local updating rule, has shown interesting features: it can be easily calibrated to be consistent with the results of commonly accepted air pollution models, it can immediately fit different environmental conditions and can be easily implemented on SIMD (Single Instruction Multiple Data) machines, thus effectively exploiting the parallelization inherent to the CA approach.

Massively parallel computational simulations in light scattering

Described is the Connection Machine, a massively parallel SIMD (single instruction multiple data) computer of 64K processors, used to simulate optical scattering and absorption phenomena where the scattering objects are much larger than the wavelength of the incident light. One processor of the connection machine is assigned to an individual incident ray. The processor tracks the ray through the object system noting the multiple reflections and refractions which occur. The results calculated are the vectors of the outgoing rays which are leaving the object system environment. As each of the interactions of each incident ray are independent of those of all other rays, 64K incident rays can be tracked at a time. The classic communication bottleneck between processors on SIMD machines is nonexistent in this case. Using a stochastic approach, a Monte Carlo technique is used whereby each ray is either reflected or refracted based on probabilities. The object is represented by a collection of small facets enabling almost any shaped object or objects to be modeled from any orientation. The limit on the complexity of the object system is limited only by the size of the host memory of the Connection Machine. The algorithm is such that it runs in linear time to the number of facets which are used to represent the system.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Transient finite element computations on 65536 processors: The connection machine

AbstractThis paper reports on our experience in solving large‐scale finite element transient problems on the Connection Machine. We begin with an overview of this massively parallel processor and emphasize the features which are most relevant to finite element computations. These include virtual processors, parallel disk I/O and parallel scientific visualization capabilities. We introduce a distributed data structure and discuss a strategy for mapping thousands of processors onto a discretized structure. The combination of the parallel data structure with the virtual processor mapping algorithm is shown to play a pivotal role in efficiently achieving massively parallel explicit computations on irregular and hybrid two‐ and three‐dimensional finite element meshes. The finite element kernels written in C*/Paris have run with success to solve several examples of linear and non‐linear dynamic simulations of large problem sizes. From these example runs, we have been able to assess in detail their performance on the Connection Machine. We show that mesh irregularities induce an MIMD (Multiple Instruction Multiple Data) style of programming which impacts negatively the performance of this SIMD (Single Instruction Multiple Data) machine. Finally, we address some important theoretical and implementational issues that will materially advance the application ranges of finite element computations on this highly parallel processor.

Coupled transport/chemistry calculations on the massively parallel processor computer

Coupled transport/chemistry problems are performed on the Massively Parallel Processor (MPP) computer. The MPP is a single-instruction multiple data (SIMD) machine with 16,384, individual processors arranged in a 2-D array with nearest neighbor connections between processors. The suitability of this architecture for coupled/transport chemistry applications is investigated. Substantial speedups can be achieved on the MPP when concurrency in the calculation can be exploited. For example, speedup factors on the MPP of 450 and 10 relative to VAX 11/780 and Cray-2, respectively, are achieved for 3-D chemistry calculation. Various aspects of parallel computing on the MPP are discussed. Transport/chemistry calculations have been performed on the NASA/Goddard MPP computer. The MPP is a SIMD machine with 16,384 individual processors arranged in a 2-D array with nearest neighbor connections between processors. This architecture is well-suited for performing chemical calculations arising from 3-D transport/chemistry problems. When operator splitting techniques are used, then each grid point in a 3-D domain can be mapped to a processor, and the chemistry at each grid point calculated in parallel. Speedup factors of over 450 and 10 were achieved on the MPP relative to the VAX 11/780 and Cray-2, respectively for the chemistry calculations. The MPP is also well-suited for 2-D transport calculations. However, due to the 2-D physical arrangement of the processor, 3-D transport problems are more difficult on the MPP. These test calculations suggest that the MPP and similar machines are well-suited for coupled transport/chemistry applications and that more robust interconnect topologies (e.g. hypercube) are needed. However, further work is necessary to gain the knowledge and experience necessary to fully exploit these machines and to evaluate their utility.

Mapping electromagnetic field computations to parallel processors

An introduction to parallel processing for the electromagnetic analyst is presented. Scaled speedup is explained, and the use of SIMD (single instruction, multiple data) and MIMD (multiple instruction, multiple data) machines is discussed. Methods of electromagnetic field computation (method of moments, finite-difference time domain, finite-element method, spectral iteration) are broadly surveyed to indicate which characteristics make them amenable to parallel processing and which architectures are best suited to those characteristics. >

A pre-design tool for a class of parallel processing systems

A properly designed parallel multi-microprocessor system can have significant speed advantage over single-microprocessor system for specific problem domains. There are several types of multi-processor systems architecture which are of interest to us. These are: 1. a. SIMD: Single Instruction Multiple Data machine (e.g., Illiac IV[1].) 2. b. MIMD: Multiple Instruction Multiple Data machine (e.g., C.mmp[2]). 3. c. MSIMD: Multiple SIMD (e.g., MAP[3]) 4. d. PASM: Partitionable SIMD/MIMD machine[4] There are a few questions to be answered for the design of such systems. How good can a multimicroprocessor system be in terms of speed and ‘best’ processing element designed for local processing? How can we re-evaluate the performance of the multi-processor system which employs different types of processing element? The question on speed has been explored by the designers though, to day, such systems, are basically particular processor oriented. The questions on local processor design and over-all evaluation are inter-related. In order to answer such questions for a multi-processor system, a simulator to evaluate the performance of the system is necessary. Simulation of a MC68000 based PASM system has been carried out at Purdue University[5]. While MC68000 is a very powerful machine, there are cases where such systems would be too powerful to be used. Also, this simulator is limited in use in the sense that it is not general purpose and only accepts MC68000 based processors as its processing element.

A Multiprocessor Architecture for the (M, L)-Algorithm Suitable for VLSI Implementation

The <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(M, L)</tex> -algorithm has been widely used in speech and image encoding. Recently, use of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(M, L)</tex> -Iike algorithms has been suggested for decoding phase codes. With its ever-increasing use, there arises a need to explore architectures suitable for real-time applications. Toward this end, we present a multiprocessor architecture for the <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(M, L)</tex> algorithm that employs an SIMD (single instruction-multiple data) machine structure. The considerations involved in interconnection network design are discussed. The main functions of the network controller are switch state selection and synchronization. The number of switching elements required is significantly less than the elements required in the universal permutation network. These features make this architecture suitable for VLSI implementation. The tradeoff between number of processors and encoding time is also discussed.

A multimicroprocessor architecture for real-time computation of a class of DFT algorithms

The design of a multimicroprocessor system intended for a class of real-time processing applications is discussed. The applications considered require the computation of a subset of an N-input Fourier transform under the assumption of a serial data input. The suitability of two different algorithms and the corresponding parallel architectures is analyzed and compared. More specifically, we compare the computation of the FFT algorithms on an SIMD (single instruction multiple data) machine to the implementation of the DFT algorithm on an MISD (multiple instruction single data) machine. Our results indicate that the latter is better suited for the targeted applications. An actual operational MISD computer, implemented with off-the-shelf microprocessors intended for one of the targeted applications, is described.