Distributed Memory Architectures Research Articles

Simulations of magneto-convection in the solar photosphere

We have developed a 3D magnetohydrodynamics simulation code for applications in the solar convection zone and photosphere. The code includes a non-local and non-grey radiative transfer module and takes into account the effects of partial ionization. Its parallel design is based on domain decomposition, which makes it suited for use on parallel computers with distributed memory architecture. We give a description of the equations and numerical methods and present the results of the simulation of a solar plage region. Starting with a uniform vertical field of 200 G, the processes of flux expulsion and convective field amplification lead to a dichotomy of strong, mainly vertical fields embedded in the granular downflow network and weak, randomly oriented fields filling the hot granular upflows. The strong fields form a magnetic network with thin, sheet- like structures extending along downflow lanes and micropores with diameters of up to 1000 km which form occasionally at vertices where several downflow lanes merge. At the visible surface around optical depth unity, the strong field concentrations are in pressure balance with their weakly magnetized surroundings and reach field strengths of up to 2 kG, strongly exceeding the values corresponding to equipartition with the kinetic energy density of the convective motions. As a result of the channelling of radiation, small flux concentrations stand out as bright features, while the larger micropores appear dark in brightness maps owing to the suppression of the convective energy transport. The overall shape of the magnetic network changes slowly on a timescale much larger than the convective turnover time, while the magnetic flux is constantly redistributed within the network leading to continuous formation and dissolution of flux concentrations.

Solving the block-Toeplitz least-squares problem in parallel

In this paper we present two versions of a parallel algorithm to solve the block–Toeplitz least-squares problem on distributed-memory architectures. We derive a parallel algorithm based on the seminormal equations arising from the triangular decomposition of the product T TT . Our parallel algorithm exploits the displacement structure of the Toeplitz-likematrices using theGeneralized SchurAlgorithm to obtain the solution in O(mn) flops instead of O(mn2) flops of the algorithms for non-structured matrices. The strong regularity of the previous product of matrices and an appropriate computation of the hyperbolic rotations improve the stability of the algorithms. We have reduced the communication cost of previous versions, and have also reduced the memory access cost by appropriately arranging the elements of the matrices. Furthermore, the second version of the algorithm has a very low spatial cost, because it does not store the triangular factor of the decomposition. The experimental results show a good scalability of the parallel algorithm on two different clusters of personal computers. Copyright c © 2005 John Wiley & Sons, Ltd.

Parallel MCGLS and ICGLS Methods for Least Squares Problems on Distributed Memory Architectures

In this paper we mainly study the parallelization of the CGLS method, a basic iterative method for large and sparse least squares problems in which the conjugate gradient method is applied to solve normal equations. On modern parallel architectures its parallel performance is always limited because of the global communication required for inner products, the main bottleneck of parallel performance. In this paper, we describe a modified CGLS (MCGLS) method which improve parallel performance by assembling the results of a number of inner products collectively and by creating situations where communication can be overlapped with computation. More importantly, we also propose an improved CGLS (ICGLS) method to reduce inner product's global synchronization points to half, then significantly improve the parallel performance accordingly compared with the standard CGLS method and the MCGLS method.

Parallel Fortran-MPI software for numerical inversion of the Laplace transform and its application to oscillatory water levels in groundwater environments

A parallel Fortran-MPI (Message Passing Interface) software for numerical inversion of the Laplace transform based on a Fourier series method is developed to meet the need of solving intensive computational problems involving oscillatory water level's response to hydraulic tests in a groundwater environment. The software is a parallel version of ACM (The Association for Computing Machinery) Transactions on Mathematical Software (TOMS) Algorithm 796. Running 38 test examples indicated that implementation of MPI techniques with distributed memory architecture speedups the processing and improves the efficiency. Applications to oscillatory water levels in a well during aquifer tests are presented to illustrate how this package can be applied to solve complicated environmental problems involved in differential and integral equations. The package is free and is easy to use for people with little or no previous experience in using MPI but who wish to get off to a quick start in parallel computing.

Inexact block Newton methods for solving nonlinear equations

In the paper two parallelizable inexact block Newton methods are presented for solving large and sparse nonlinear equations. The basic idea is simple and direct. Combining the simplified Newton method with the component averaging (CAV) method [Parallel Comput. 27 (2001) 777] results in an inexact Newton method, called simplified Newton-CAV method. Parallel tests of the algorithm are implemented on a message-passing distributed-memory multiprocessor architecture such as a cluster of workstations. The results show that the new algorithm can achieve good performance. Moreover as a development of the simplified Newton-CAV method, the overlapped block Newton-CAV method is further proposed and discussed.

Design and evaluation of a TOP100 Linux Super Cluster system

AbstractThe High Performance Computing Center North (HPC2N) Super Cluster is a truly self‐made high‐performance Linux cluster with 240 AMD processors in 120 dual nodes, interconnected with a high‐bandwidth, low‐latency SCI network. This contribution describes the hardware selected for the system, the work needed to build it, important software issues and an extensive performance analysis. The performance is evaluated using a number of state‐of‐the‐art benchmarks and software, including STREAM, Pallas MPI, the Atlas DGEMM, High‐Performance Linpack and NAS Parallel benchmarks. Using these benchmarks we first determine the raw memory bandwidth and network characteristics; the practical peak performance of a single CPU, a single dual‐node and the complete 240‐processor system; and investigate the parallel performance for non‐optimized dusty‐deck Fortran applications. In summary, this $500 000 system is extremely cost‐effective and shows the performance one would expect of a large‐scale supercomputing system with distributed memory architecture. According to the TOP500 list of June 2002, this cluster was the 94th fastest computer in the world. It is now fully operational and stable as the main computing facility at HPC2N. The system's utilization figures exceed 90%, i.e. all 240 processors are on average utilized over 90% of the time, 24 hours a day, seven days a week. Copyright © 2004 John Wiley & Sons, Ltd.

Simulation of Sheared Suspensions With a Parallel Implementation of QDPD.

A parallel quaternion-based dissipative particle dynamics (QDPD) program has been developed in Fortran to study the flow properties of complex fluids subject to shear. The parallelization allows for simulations of greater size and complexity and is accomplished with a parallel link-cell spatial (domain) decomposition using MPI. The technique has novel features arising from the DPD formalism, the use of rigid body inclusions spread across processors, and a sheared boundary condition. A detailed discussion of our implementation is presented, along with results on two distributed memory architectures. A parallel speedup of 24.19 was obtained for a benchmark calculation on 27 processors of a distributed memory cluster.

LLC: A PARALLEL SKELETAL LANGUAGE

The skeletal approach to the development of parallel applications has been revealed to be one of the most successful and has been widely explored in the recent years. The goal of this approach is to develop a methodology of parallel programming based on a restricted set of parallel constructs. This paper presents llc, a parallel skeletal language, the theoretical model that gives support to the language and a prototype implementation for its compiler. The language is based on directives, uses a C-like syntax and gives support to the most widely used skeletal constructs. llCoMP is a source to source compiler for the language built on top of MPI. We evaluate the performance of our prototype compiler using four different parallel architectures and three algorithms. We present the results obtained in both shared and distributed memory architectures. Our model guarantees the portability of the language to any platform and its simplicity greatly eases its implementation.

Parallel Genehunter: implementation of a linkage analysis package for distributed-memory architectures

We present a parallel algorithm for performing multipoint linkage analysis of genetic marker data on large family pedigrees. The algorithm effectively distributes both the computation and memory requirements of the analysis. We discuss an implementation of the algorithm in the Genehunter linkage analysis package (version 2.1), enabling Genehunter to run on distributed-memory platforms for the first time. Our preliminary benchmarks indicate reasonable scalability of the algorithm even for fixed-size problems, with parallel efficiencies of 75% or more on up to 128 processors. In addition, we have extended the hard-coded limit of 16 non-founding individuals in Genehunter 2.1 to a new limit of 32 non-founding individuals.

A new efficient approach to the direct restricted active space self-consistent field method

We present an implicitly parallel method for integral-block driven restricted active space self-consistent field (RASSCF) algorithms. Our algorithm entirely avoids testing the index space for nonzero contributions to the CI vector, by finding entire blocks of contributions through use of simple algebraic rules (propagation rules). The blocks themselves are efficiently identified by introducing a RAS model space. Our algorithm is capable of making efficient use of modern supercomputer hardware, supporting both shared and distributed memory architectures and hybrids. Applicability of our method is demonstrated with a RASSCF investigation of the first two excited states of indole.

Parallelization of a treecode

I describe here the performance of a parallel treecode with individual particle timesteps. The code is based on the Barnes–Hut algorithm and runs cosmological N-body simulations on parallel machines with a distributed memory architecture using the MPI message-passing library. For a configuration with a constant number of particles per processor the scalability of the code was tested up to P=128 processors on an IBM SP4 machine. In the large P limit the average CPU time per processor necessary for solving the gravitational interactions is ∼10% higher than that expected from the ideal scaling relation. The processor domains are determined every large timestep according to a recursive orthogonal bisection, using a weighting scheme which takes into account the total particle computational load within the timestep. The results of the numerical tests show that the load balancing efficiency L of the code is high (≳90%) up to P=32, and decreases to L∼80% when P=128. In the latter case it is found that some aspects of the code performance are affected by machine hardware, while the proposed weighting scheme can achieve a load balance as high as L∼90% even in the large P limit.

Parallel iterative solution of large‐scale acoustic scattering problems using exact non reflecting conditions on distributed memory computer systems

Parallel iterative methods for fast solution of large‐scale acoustic radiation and scattering problems are developed using exact Dirichlet‐to‐Neumann (DtN) nonreflecting boundaries. For elongated scatterers such as submarines, it is shown that the generalization of the DtN to elliptical/spheroidal artificial boundaries improves significantly the computational efficiency of accurate finite element methods for the solution of acoustic scattering problems. The outer‐product structure of the DtN map is exploited as a low‐rank update of the system matrix to efficiently compute the matrix‐by‐vector products found in Krylov subspace based iterative methods. For the complex non‐Hermitian matrices resulting from the Helmholtz equation, a distributed‐memory parallel BICG‐STAB iterative method is used in conjunction with a hybrid parallel SSOR/Jacobi preconditioner. The domain decomposition with interface minimization was performed to ensure optimal inter‐processor communication. For the distributed memory architectures tested, including Linux/Intel Beowulf clusters, when implemented as a low‐rank update, the nonlocal character of the DtN map shows little impact on the scale up or parallel efficiency compared to approximate local boundary conditions. [Work supported by NSF.]

A highly flexible, distributed multiprocessor architecture for network processing

Network processors (NPs) are an emerging field of programmable processors that are optimized to implement data plane packet processing networking functions. Unlike the general-purpose CPUs that rely heavily on caching for improving performance, the lack of locality in packet processing and need for high-performance I/O have forced designers to come up with innovative architectures that can hide memory latency while still processing packets at high data rates. Most of these NPs use some type of multiprocessing in combination with a hierarchy of memory types to achieve high performance. In addition, to keep up with packets arriving at high data rates over multiple incoming media interfaces, an NP must perform fast I/O and memory operations such as packet storage, table lookup, and extraction of fields in packet headers. We describe an architecture that uses a combination of distributed memory architecture and one or more multithreaded processors to achieve the necessary performance. We describe the challenges in programming such a processor including the issues related to consistency and maintaining packet ordering. We also present a programming model for generic network applications that uses software pipelines. We then demonstrate the use of the programming model in implementing two applications, namely, mapping traffic management algorithms onto a multithreaded architecture and an implementation of a media gateway based on voice-over-AAL2.

Non-strict execution in parallel and distributed computing

This paper surveys and demonstrates the power of non-strict evaluation in applications executed on distributed architectures. We present the design, implementation, and experimental evaluation of single assignment, incomplete data structures in a distributed memory architecture and Abstract Network Machine (ANM). Incremental Structures (IS), Incremental Structure Software Cache (ISSC), and Dynamic Incremental Structures (DIS) provide nonstrict data access and fully asynchronous operations that make them highly suited for the exploitation of fine-grain parallelism in distributed memory systems. We focus on split-phase memory operations and non-strict information processing under a distributed address space to improve the overall system performance. A novel technique of optimization at the communication level is proposed and described. We use partial evaluation of local and remote memory accesses not only to remove much of the excess overhead of message passing, but also to reduce the number of messages when some information about the input or part of the input is known. We show that split-phase transactions of IS, together with the ability of deferring reads, allow partial evaluation of distributed programs without losing determinacy. Our experimental evaluation indicates that commodity PC clusters with both IS and a caching mechanism, ISSC, are more robust. The system can deliver speedup for both regular and irregular applications. We also show that partial evaluation of memory accesses decreases the traffic in the interconnection network and improves the performance of MPI IS and MPI ISSC applications.

Parallel Performance Investigations of an Unstructured Mesh Navier-Stokes Solver

Summary The implementation and performance of a hybrid OpenMP/ MPI parallel communication strategy for an unstructured mesh computational fluid dynamics code is described. The solver is cache efficient and fully vectorizable, and is parallelized using a two-level hybrid MPI-OpenMP implementation suitable for shared and/or distributed memory architectures, as well as clusters of shared memory machines. Parallelism is obtained through domain decomposition for both communication models. Single processor computational rates as well as scalability curves are given on various architectures. For the architectures studied in this work, the OpenMP or hybrid OpenMP/MPI communication strategies achieved no appreciable performance benefit over an exclusive MPI communication strategy.

A Hybrid Mpi-Openmp Implementation of an Implicit Finite-Element Code on Parallel Architectures

Summary The hybrid MPI-OpenMP model is a natural parallel programming paradigm for emerging parallel architectures that are based on symmetric multiprocessor (SMP) clusters. This paper presents a hybrid implementation adapted for an implicit finite-element code developed for groundwater transport simulations. The original code was parallelized for distributed memory architectures using MPI (Message Passing Interface) using a domain decomposition strategy. OpenMP directives were then added to the code (a straightforward loop-level implementation) to use multiple threads within each MPI process. To improve the OpenMP performance, several loop modifications were adopted. The parallel performance results are compared for four modern parallel architectures. The results show that for most of the cases tested, the pure MPI approach outperforms the hybrid model. The exceptions to this observation were mainly due to a limitation in the MPI library implementation on one of the architectures. A general conclusion is that while the hybrid model is a promising approach for SMP cluster architectures, at the time of this writing, the payoff may not be justified for converting all existing MPI codes to hybrid codes. However, improvements in OpenMP compilers combined with potential MPI limitations in SMP nodes may make the hybrid approach more attractive for a broader set of applications in the future.

Time-domain Laplace transform boundary element methods for diffusion problems

In the solution of diffision-~e problems the Laplace transform method in time provides an alternative to the finite difference method. The advantages compared with the finite difference method are that there is no time-step stability problem and if the solution is required at just one time value then there is no need for the computation of solutions at intermediate times. From a variety of numerical techniques available for the numerical inversion, Stehfest’s method has been the numerical inversion process chosen by most authors in association with the boundary element method. The procedure has been successfidly implemented on a distributed memory architecture using a ‘Single Program Multiple Data’ paradigm on a 512-processor nCUBE machine arranged in a hypercube configuration. The Laplace transform method is applicable to linear problems, for non-linear problems a suitable linearisation process is required.

An experimental evaluation of I/O optimizations on different applications

Many large-scale applications have significant I/O requirements as well as computational and memory requirements. Unfortunately, the limited number of I/O nodes provided in a typical configuration of the modern message-passing distributed-memory architectures such as the Intel Paragon and the IBM SP-2 limits the I/O performance of these applications severely. In this paper, we examine some software optimization techniques and evaluate their effects in five different I/O-intensive codes from both small and large application domains. Our goals in this study are twofold. First, we want to understand the behavior of large-scale data-intensive applications and the impact of I/O subsystems on their performance and vice versa. Second, and more importantly, we strive to determine the solutions for improving the applications' performance by a mix of software techniques. Our results reveal that different applications can benefit from different optimizations. For example, we found that some applications benefit from file layout optimizations, whereas others take advantage of collective I/O. A combination of architectural and software solutions is normally needed to obtain good I/O performance. For example, we show that with a limited number of I/O resources, it is possible to obtain good performance by using appropriate software optimizations. We also show that beyond a certain level, imbalance in the architecture results in performance degradation even when using optimized software, thereby indicating the necessity of an increase in I/O resources.

Full-field measurements of heterogeneous deformation patterns on polymeric foams using digital image correlation

The ability of a digital image correlation technique to capture the heterogeneous deformation fields appearing during compression of ultra-light open-cell foams is presented in this article. Quantitative characterization of these fields is of importance to understand the mechanical properties of the collapse process and the energy dissipation patterns in this type of materials. The present algorithm is formulated in the context of multi-variable non-linear optimization where a merit function based on a local average of the deformation mapping is minimized implicitly. A parallel implementation utilizing message passing interface for distributed-memory architectures is also discussed. Estimates for optimal size of the correlation window based on measurement accuracy and spatial resolution are provided. This technique is employed to reveal the evolution of the deformation texture on the surface of open-cell polyurethane foam samples of different relative densities. Histograms of the evolution of surface deformation are extracted, showing the transition from unimodal to bimodal and back to unimodal. These results support the interpretation that the collapse of light open-cell foams occurs as a phase transition phenomenon.

A parallel Monte Carlo code for planar and SPECT imaging: implementation, verification and applications in 131I SPECT

This paper reports the implementation of the SIMIND Monte Carlo code on an IBM SP2 distributed memory parallel computer. Basic aspects of running Monte Carlo particle transport calculations on parallel architectures are described. Our parallelization is based on equally partitioning photons among the processors and uses the Message Passing Interface (MPI) library for interprocessor communication and the Scalable Parallel Random Number Generator (SPRNG) to generate uncorrelated random number streams. These parallelization techniques are also applicable to other distributed memory architectures. A linear increase in computing speed with the number of processors is demonstrated for up to 32 processors. This speed-up is especially significant in Single Photon Emission Computed Tomography (SPECT) simulations involving higher energy photon emitters, where explicit modeling of the phantom and collimator is required. For 131I, the accuracy of the parallel code is demonstrated by comparing simulated and experimental SPECT images from a heart/thorax phantom. Clinically realistic SPECT simulations using the voxel-man phantom are carried out to assess scatter and attenuation correction.