Cray T3D Research Articles

HIGH-SPEED ISING MONTE CARLO AND ELASTIC PARTICLE MOLECULAR DYNAMICS SIMULATIONS

The performance of Cray T3D and some other processors was measured with FORTRAN programs for two kinds of statistical mechanical simulation. One is the Monte Carlo simulation of ferromagnetic Ising model on square and cubic lattices with up to 1010 updates per second. The other is the molecular dynamics simulation for two-dimensional elastic disk system.

Compiling Array Expressions for Efficient Execution on Distributed-Memory Machines

Array statements are often used to express data-parallelism in scientific languages such as Fortran 90 and High Performance Fortran. In compiling array statements for a distributed-memory machine, efficient generation of communication sets and local index sets is important. We show that for arrays distributed block-cyclically on multiple processors, the local memory access sequence and communication sets can be efficiently enumerated as closed forms using regular sections. First, closed form solutions are presented for arrays that are distributed using block or cyclic distributions. These closed forms are then used with avirtual processor approachto give an efficient solution for arrays with block-cyclic distributions. This approach is based on viewing a block-cyclic distribution as a block (or cyclic) distribution on a set of virtual processors, which are cyclically (or block-wise) mapped to physical processors. These views are referred to asvirtual-blockorvirtual-cyclicviews, depending on whether a block or cyclic distribution of the array on the virtual processors is used. The virtual processor approach permits different schemes based on the combination of the virtual processor views chosen for the different arrays involved in an array statement. These virtualization schemes have different indexing overhead. We present a strategy for identifying the virtualization scheme which will have the best performance. Performance results on a Cray T3D system are presented for hand-compiled code for array assignments. These results show that using the virtual processor approach, efficient code can be generated for execution of array statements involving block-cyclically distributed arrays.

Parallelization of and Results from Kinetic Edge Plasma Code W1

AbstractIn this paper, we describe a highly parallelized implementation of the 1D2V collisional PIC‐MCC code (W1 by Batishchev[1]) for Scrape‐Off‐Layer (SOL) plasmas on the Cray T3D using the PVM message‐passing library. The particles are divided equally among the processors, and grid quantities (temperatures, etc) are averaged from all processors. The electric field is solved using the electron momentum equation. The logical sheath potential is calculated in a serial fashion. For a fixed number of particles in the simulations (1 million particles), we have obtained a factor of 31 speed‐up by using 32 processors compared to 1, and a factor of 90 speed‐up by using 128 processors. Through our efforts on Cray T3D with 256 processors we have learned that massively parallel MIMD systems offers a new supercomputing capability for kinetic SOL physics modelling.

The Pittsburgh Supercomputing Center

Since its inception in 1986, the Pittsburgh Supercomputing Center has used high performance computing to solve significant problems in biomedical engineering, oceanography, astronomy, climate and weather forecasting, and a host of other disciplines. To date, more than 10,000 researchers from more than 1,300 universities and research centers in 49 states have used the Center's facilities. This work has resulted in more than 3,000 published papers. PSC's computing projects use three main production machines: the Cray C90, a vector multiprocessor; the Cray T3D, a scalable, massively parallel processing system; and the DEC Alpha Supercluster, a workstation cluster. The Center concentrates on capability computing, which provides scheduling flexibility and technical and scientific expertise while encouraging single users to exploit maximum processing potential over sustained, uninterrupted periods. This means using all 16 processors on the Cray C90, or all 512 processors on the Cray T3D, for a single project.

Perspectives on supercomputing: three decades of change

I am fortunate to have had access to supercomputers for the last 28 years. Over this time I have used them to simulate time-dependent fluid flows in the compressible regime. Strong shocks and unstable multifluid boundaries, along with the phenomenon of fluid turbulence, have provided the simulation complexity that demands supercomputer power. The supercomputers I have used-the CDC 6600, 7600, and Star-100, the Cray-1, Cray-XMP, Cray-2, and Cray C-90, the Connection Machines CM-2 and CM-5, the Cray T3D, and the Silicon Graphics Challenge Array and Power Challenge Array-span three revolutions in supercomputer design: the introduction of vector supercomputing, parallel supercomputing on multiple CPUs, and supercomputing on hierarchically organized clusters of microprocessors with cache memories. The last revolution is still in progress, so its outcome is somewhat uncertain. I view these design revolutions through the prism of my specialty and through applications of the supercomputers I have used. Also, because these supercomputer design changes have driven equally important changes in numerical algorithms and the programs that implement them, I describe the three revolutions from this perspective.

Parallel programming with Polaris

Parallel programming tools are limited, making effective parallel programming difficult and cumbersome. Compilers that translate conventional sequential programs into parallel form would liberate programmers from the complexities of explicit, machine oriented parallel programming. The paper discusses parallel programming with Polaris, an experimental translator of conventional Fortran programs that target machines such as the Cray T3D.

Global arrays: A nonuniform memory access programming model for high-performance computers

Portability, efficiency, and ease of coding are all important considerations in choosing the programming model for a scalable parallel application. The message-passing programming model is widely used because of its portability, yet some applications are too complex to code in it while also trying to maintain a balanced computation load and avoid redundant computations. The shared-memory programming model simplifies coding, but it is not portable and often provides little control over interprocessor data transfer costs. This paper describes an approach, called Global Arrays (GAs), that combines the better features of both other models, leading to both simple coding and efficient execution. The key concept of GAs is that they provide a portable interface through which each process in a MIMD parallel program can asynchronously access logical blocks of physically distributed matrices, with no need for explicit cooperation by other processes. We have implemented the GA library on a variety of computer systems, including the Intel Delta and Paragon, the IBM SP-1 and SP-2 (all message passers), the Kendall Square Research KSR-1/2 and the Convex SPP-1200 (nonuniform access shared-memory machines), the CRAY T3D (a globally addressable distributed-memory computer), and networks of UNIX workstations. We discuss the design and implementation of these libraries, report their performance, illustrate the use of GAs in the context of computational chemistry applications, and describe the use of a GA performance visualization tool.

PARAMICS?Parallel microscopic simulation of road traffic

This paper describes work done on the original PARAMICS project, which was developed for the Edinburgh Parallel Computing Centre to examine parallel microscopic road traffic simulation. The simulator, constructed originally for a Thinking Machines Connection Machine (CM-200), uses a data-parallel approach to simulate approximately 200,000 vehicles on 20,000 miles of roadway. More recent work has focused on the use of a message-passing paradigm, with a 256-node CRAY T3D as the target machine. The message-passing version of PARAMICS, PARAMICS-MP, is inherently scalable and can model many smaller networks on a broad range of platforms.

Asynchronous Massively Parallel Genetic Algorithm Using Artificial-Society Model and Its Application.

Asynchronous massively parallel genetic algorithm (AMPGA) based on a newly proposed society was implemented for relatively large scale optimization problems; and its parallel processing efficiency was evaluated using several optimization problems such as shape optimization and traveling salesman problems in this paper. Because of its asynchronous characteristics produced by the artificial society model which is an agent-oriented model, an emergent behavior such as timings of mating for crossover/mutation and even intergenerational crossover was also observed through shape optimization problems. For traveling salesman problem, large population model was also included in the proposed AMPGA and the effect of the number of agents on each processor was observed in this study. As a result, two types of shape optimization problems and 105-city traveling salesman problem were tested on CRAY T3D massively parallel processor system using the proposed method, and its applicability and parallel processing efficiency were verified.

A CLUSTERING ALGORITHM FOR PARALLEL SPARSE CHOLESKY FACTORIZATION

This paper presents an integrated approach to two issues relevant to efficient parallel sparse Cholesky factorization: 1) matrix reordering for parallelism, and, 2) mapping of data to processors. A clustering heuristic is proposed to performs a fill-preserving reordering and mapping of data onto a fixed number of processors. Performance results on a Cray T3D are presented to demonstrate its effectiveness.

A Comparison of Parallel Programming Models for Multiblock Flow Computations

A study is presented of the implementation of four different parallel programming models in a code that solves the fluid flow equations on block structured meshes. Performance results obtained on a number of distributed-memory parallel computer systems are given, in particular, for a 1024 processor Cray T3D system. Using the appropriate programming model, it is shown that excellent performance scaling can be obtained even for small problem sizes. The relative merits of each programming model in terms of ease of use, functionality, and performance are assessed.

Massively parallel semi-Lagrangian advection

The passive advection problem in computational fluid dynamics is examined. Eulerian methods are restricted by the Courant-Friedrichs-Lewy (CFL) condition and the semi-Lagrangian method is introduced as an alternative approach for taking longer time steps. Recent progress in the development of parallel algorithms for semi-Lagrangian advection is reviewed and implementation strategies for distributed MIMD computation are discussed. Predicted and actual parallel performance results on the Intel iPSC/860 and the Cray T3D MPP are presented and it is established that the problem is scalable.

A highly portable parallel implementation of AMBER4 using the message passing interface standard

AbstractWe have implemented a portable parallel version of the macromolecular modeling package AMBER4. The message passing paradigm was used. All message passing constructs are compliant with the Message Passing Interface (MPI) standard. The molecular dynamics/minimization module MINMD and the free‐energy perturbation module Gibbs have been implemented in parallel on a number of machines, including a Cray T3D, an IBM SP1/SP2, and a collection of networked workstations. In addition, the code has been tested with an MPI implementation from Argonne National Laboratories/Mississippi State University which runs on many parallel machines. The goal of this work is to decrease the amount of time required to perform molecular dynamics simulations. Performance results for a lipid bilayer molecular dynamics simulation on a Cray T3D, an IBM SP1/SP2, and a Cray C90 are compared. © 1995 John Wiley & Sons, Inc.

A comparison of data-parallel and message-passing versions of the Miami Isopycnic Coordinate Ocean Model (MICOM)

A two-pronged effort to convert a recently developed ocean circulation model written in Fortran-77 for execution on massively parallel computers is described. A data-parallel version was developed for the CM-5 manufactured by Thinking Machines, Inc., while a message-passing version was developed for both the Cray T3D and the Silicon Graphics ONYX workstation. Since the time differentiation scheme in the ocean model is fully explicit and does not require solution of elliptic partial differential equations, adequate machine utilization has been achieved without major changes to the original algorithms. We developed a partitioning strategy for the message passing version that significantly reduces memory requirements and increases model speed. On a per-node basis (a T3D node is one Alpha processor, a CM-5 node is one Sparc chip and four vector units), the T3D and CM-5 are found to execute our “large” model version consisting of 511 × 511 horizontal mesh points at roughly the same speed.

Weather forecasting on parallel architectures

Global forecasting at the National Meteorological Center is performed using a spherical harmonics spectral model. The model is used in global data assimilation, and in the aviation and medium range forecast production suites. In this presentation, the major numerical and physical aspects of the model are highlighted, and parallelization considerations are analyzed and applied to three parallel computer architectures. The first parallelization is on a parallel vector processor shared memory platform such as Cray YMP or C90. The other two are on distributed memory platforms: a data parallel implementation on Connection Machines and a single program multiple data (SPMD) adaptation to a Cray T3D machine.

PVM‐AMBER: A parallel implementation of the AMBER molecular mechanics package for workstation clusters

AbstractA parallel version of the popular molecular mechanics package AMBER suitable for execution on workstation clusters has been developed. Computer‐intensive portions of molecular dynamics or free‐energy perturbation computations, such as nonbonded pair list generation or calculation of nonbonded energies and forces, are distributed across a collection of Unix workstations linked by Ethernet or FDDI connections. This parallel implementation utilizes the message‐passing software PVM (Parallel Virtual Machine) from Oak Ridge National Laboratory to coordinate data exchange and processor synchronization. Test simulations performed for solvated peptide, protein, and lipid bilayer systems indicate that reasonable parallel efficiency (70–90%) and computational speedup (2–5 × serial computer runtimes) can be achieved with small workstation clusters (typically six to eight machines) for typical biomolecular simulation problems. PVM‐AMBER is also easily and rapidly portable to different hardware platforms due to the availability of PVM for numerous computers. The current version of PVM‐AMBER has been tested successfully on Silicon Graphics, IBM RS6000, DEC ALPHA, and HP 735 workstation clusters and heterogeneous clusters of these machines, as well as on CRAY T3D and Kendall Square KSR2 parallel supercomputers. Thus, PVM‐AMBER provides a simple and cost‐effective mechanism for parallel molecular dynamics simulations on readily available hardware platforms. Factors that affect the efficiency of this approach are discussed. © 1995 by John Wiley & Sons, Inc.

A PIPELINED BROADCAST FOR MULTIDIMENSIONAL MESHES

We address the problem of performing a pipelined broadcast on a mesh architecture. Meshes require a different approach than other topologies, and their very nature puts a tighter bound on the performance that one can hope to achieve. By using the appropriate techniques, however, one can obtain excellent performance for sufficiently long messages. The resulting algorithm will work on meshes of any dimension with any number of nodes. Our model assumes that the mesh is a torus and/or that it has bidirectional links and uses wormhole routing. Performance data from the Cray T3D are included.

Dynamic load balancing algorithms for replicated data molecular dynamics

AbstractAlgorithms to enhance parallel performance of molecular dynamics simulations on parallel computers by dynamic load balancing are described. Load balancing is achieved by redistribution of work based on either a history of time spent computing per processor or on the number of pair interactions computed per processor. The two algorithms we detail are designed to yield optimal load balancing on both workstation clusters and parallel supercomputers. We illustrate these methods using a small molecular dynamics kernel developed for the simulation of rigid molecular solvents. In addition, we discuss our observation regarding global communications performance on workstation clusters with a fiber distributed data interface (FDDI) using a high‐speed point‐to‐point switch (Gigaswitch) and the k‐ary 3‐cube of the Cray T3D. © 1995 by John Wiley & Sons, Inc.

A comparison of architectural support for messaging in the TMC CM-5 and the Cray T3D

Programming models based on messaging continue to be an important programming model for parallel machines. Messaging costs are strongly influenced by a machine's network interface architecture. We examine the impact of architectural support for messaging in two machines --- the TMC CM-5 and the Cray T3D --- by exploring the design and performance of several messaging implementations. The additional features in the T3D support remote operations: memory access, fetch-and-increment, atomic swaps, and prefetch.Experiments on the CM-5 show that requiring processor involvement for message reception can increase the communication overheads from 60% to 300% for moderate variations in computation grain size at the destination. In contrast, the T3D hardware for remote operations decouples message reception from processor activity, producing high-performance messaging independent of computation grain size or variability.In addition, hardware support for a shared address space in the T3D can be used to solve the output contention problem (output hot spots), producing messaging implementations that are robust over a wide variety of traffic patterns. Atomic swap hardware can be used to build a distributed message queue, enabling a "pull" messaging scheme where the destination requests data transfer upon receive. This scheme uses prefetches to mask receive latency. While this yields performance robust over output contention, its base cost is competitive only for small messages (up to 64 bytes) because of the high cost of issuing and resolving prefetches in the T3D. Emulation shows that if the interaction costs can be reduced by a factor of eight (250ns to 31ns), perhaps by moving the prefetch queue on chip, and there is a corresponding increase in the prefetch queue size, the pull scheme can give superior performance in all eases.

On characterizing bandwidth requirements of parallel applications

Synthesizing architectural requirements from an application viewpoint can help in making important architectural design decisions towards building large scale parallel machines. In this paper, we quantify the link bandwidth requirement on a binary hypercube topology for a set of five parallel applications. We use an execution-driven simulator called SPASM to collect data points for system sizes that are feasible to be simulated. These data points are then used in a regression analysis for projecting the link bandwidth requirements for larger systems. The requirements are projected as a function of the following system parameters: number of processors, CPU clock speed, and problem size. These results are also used to project the link bandwidths for other network topologies. Our study quantifies the link bandwidth that has to be made available to limit the network overhead in an application to a specified tolerance level. The results show that typical link bandwidths (200-300 MBytes/sec) found in current commercial parallel architectures (such as Intel Paragon and Cray T3D) would have fairly low network overhead for the applications considered in this study. For two of the applications, this overhead is negligible. For the other applications, this overhead can be limited to about 30% of the execution time provided the problem sizes are increased commensurate with the processor clock speed. The technique presented can be useful to a system architect to synthesize the bandwidth requirements for realizing well-balanced parallel architectures.