Distributed Memory Computers Research Articles

The VCI-P code: an iterative variation–perturbation scheme for efficient computations of anharmonic vibrational levels and IR intensities of polyatomic molecules

A variation–perturbation scheme called VCI-P code is proposed. It is developed mainly for the efficient and accurate theoretical description of vibrational resonances in polyatomic molecules. This state-specific process consists of the iterative construction of small matrices for each vibrational state, using the most important configurations contributing to that state. The weak couplings are considered perturbationally. The keypoint of this paper is a recipe that allows the massive truncation of the vibrational configuration space with minimum error in the calculated energies. The anharmonic frequencies and IR/vibrational absorption (VA) intensities obtained using VCI-P for methane and formaldehyde are compared to their full VCI counterparts. The convergence of the VCI-P results with respect to configuration selection is also discussed from the examples of trans-difluoroethylene and ethylene oxide (also called oxirane). A parallelization scheme for the 3N − 5 calculations on distributed memory computers is proposed. Representative computational times are presented for molecules ranging in size from 4 to 15 atoms.

An Implementation of Parallel Fast Marching Using the Message Passing Interface

This document introduces a program based on the algorithm described by Maria Cristina Tugurlan. The program uses file readers, image filters, and file writers from the Insight Toolkit ITK www.itk.org. It produces as output an image whose values are the times of first arrival of a wavefront that spreads from seed points with a speed at every point equal to the input image intensity. It performs the computation in parallel on distributed memory computers using the Message Passing Interface MPI. Each MPI process reads a small piece of the input image into memory. It computes fast marching on its piece. It sends and receives the values from fast marching at piece boundaries. It recalculates fast marching a number of times set from the command line using the new boundary values each time. Each MPI process writes only a small piece of the output file.A substantial difference is seen when the output of MPI fast marching is compared to the output of serial fast marching. This difference may be acceptable for some uses. The program should be able to handle input images that are too large to fit in the memory of a single computer.

Parallel implementation of the ADI‐FDTD method

AbstractWe present a parallel implementation of the three‐dimensional alternating direction implicit finite‐difference time‐domain (ADI‐FDTD) method in Cartesian coordinates using the message passing interface (MPI) library. Parallel implementations not only speed up computations but also increase the maximum solvable problem size. The application of the ADI scheme results in freeing the time‐step size from the Courant‐Friedrichs‐Lewy stability constraint. We demonstrate a speedup of the ADI‐FDTD method through parallelization, even though the communication overhead between the processors is proportional to the volume of the domain, rather than just the surface area of the subdomains as for standard parallel FDTD. We benchmarked our code on an IBM p690 symmetric multiprocessor and also a distributed memory computer cluster with a high‐bandwidth Infiniband interconnect. In both cases, satisfactory scalability and efficiency was demonstrated for simulation domains comprising up to 8 billion mesh cells. © 2009 Wiley Periodicals, Inc. Microwave Opt Technol Lett 51: 1298–1304, 2009; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/mop.24310

Superlinear Speedup in a 3-D Parallel Conjugate Gradient Solver

This paper presents a computational performance analysis of a parallel implementation of a conjugate gradient (CG) solver using domain decomposition and distributed memory computers, applied to a 3-D finite element method problem. The results show a superlinear speedup, which is not usually expected. The analysis shows why and how it can happen.

Ten organising principles for coupling in multiphysics and multiscale models

Computational science faces new challenges posed by multiphysics and multiscale, or more generally put, coupled models. These systems are composites formed from separate subsystem models that interact via data exchanges. These data dependencies pose a coupling problem, and on distributed memory computers, a parallel coupling problem. This article presents a definition of terms and a set of organising principles for the coupling and parallel coupling problems. It is meant as a first step towards creating a theory of coupled models. These principles are then employed in a case study of a coupled climate model and offer remarkable insight into its structure. References J. R. Cary, J. Candy, R. H. Cohen, S. Krasheninnikov, D. C. McCune, D. J. Estep, J. Larson, A. D. Malony, P. H. Worley, J. A. Carlsson, A. H. Hakim, P. Hamill, S. Kruger, S. Muzsala, A. Pletzer, S. Shasharina, D. Wade-Stein, N. Wang, L. McInnes, T. Wildey, T. Casper, L. Diachin, T. Epperly, T. D. Rognlien, M. R. Fahey, J. A. Kuehn, A. Morris, S. Shende, E. Feibush, G. W. Hammett, K. Indireshkumar, C. Ludescher, L. Randerson, D. Stotler, A. Yu Pigarov, P. Bonoli, C. S. Chang, D. A. D'Ippolito, P. Colella, D. E. Keyes, R. Bramley, and J. R. Myra. Introducing facets, the framework application for core-edge transport simulations. Journal of Physics Conference Series, 78:0120086, 2007. W. D. Collins, C. M. Bitz, M. L. Blackmon, G. B . Bonan, C. S. Bretherton, J. A. Carton, P. Chang, S. C. Doney, J. J. Hack, T. B. Henderson, J. T. Kiehl, W. G. Large, D. S. McKenna, B. D. Santer, and R. D. Smith. The community climate system model: CCSM3. Journal of Climate, 19(11):2122--2143, 2006. Anthony P. Craig, Brian Kaufmann, Robert Jacob, Tom Bettge, Jay Larson, Everest Ong, Chris Ding, and Helen He. cpl6: The new extensible high-performance parallel coupler for the community climate system model. Int. J. High Perf. Comp. App., 19(3):309--327, 2005. R. Diestel. Graph Theory. Springer, New York, third edition, 2006. Ian Foster. Designing and Building Parallel Programs: Concepts and Tools for Parallel Software Engineering. Addison Wesley, Reading, Massachusetts, 1995. CCSM Software Engineering Group. CCSM research tools: CCSM3.0 coupler v6.0 documentation. http://www.ccsm.ucar.edu/models/ccsm3.0/cpl6/, 2004. W. Joppich, M. Kurschner, and the MpCCI Team. MpCCI---a Tool for the Simulation of Coupled Applications. Concurrency and Computation: Practice and Experience, 18(2):183--192, 2006. doi:10.1002/cpe.913 Jay Larson, Robert Jacob, and Everest Ong. The model coupling toolkit: A new Fortran90 toolkit for building multi-physics parallel coupled models. Int. J. High Perf. Comp. App., 19(3):277--292, 2005. WRF Development Team. Weather research and forecasting (WRF) model web site. http://wrf-model.org/, 2006.

A composite grid solver for conjugate heat transfer in fluid–structure systems

We describe a numerical method for modeling temperature-dependent fluid flow coupled to heat transfer in solids. This approach to conjugate heat transfer can be used to compute transient and steady state solutions to a wide range of fluid–solid systems in complex two- and three-dimensional geometry. Fluids are modeled with the temperature-dependent incompressible Navier–Stokes equations using the Boussinesq approximation. Solids with heat transfer are modeled with the heat equation. Appropriate interface equations are applied to couple the solutions across different domains. The computational region is divided into a number of sub-domains corresponding to fluid domains and solid domains. There may be multiple fluid domains and multiple solid domains. Each fluid or solid sub-domain is discretized with an overlapping grid. The entire region is associated with a composite grid which is the union of the overlapping grids for the sub-domains. A different physics solver (fluid solver or solid solver) is associated with each sub-domain. A higher-level multi-domain solver manages the entire solution process.We propose and analyze some centered discrete approximations to the interface equations that have some desirable stability properties. The coupled interface equations may be solved directly when using explicit time-stepping methods in the sub-domains, resulting in a strongly coupled approach. The stability of the interface treatment in this case is independent of the relative sizes of the material properties in the two domains with the time-step only depending on the usual von Neumann conditions for each sub-domain. For implicit time-stepping methods we solve the interface equations in a weakly coupled fashion to avoid forming a coupled implicit system across all sub-domains. The convergence of this approach does depend on the relative sizes of the thermal conductivities and diffusivities. We analyze different iteration strategies for solving these implicit equations including the use of mixed (Robin) approximations at the interface.Numerical results are presented to illustrate the method. The accuracy of the technique is verified using the method of analytic solutions and by computing the solution to some heat exchanger problems where the exact solution is known. The technique is also applied to the modeling of an inertial-confinement-fusion hohlraum target and the flow of coolant past an hexagonal array of heated fuel rods. The multi-domain solver runs in parallel on distributed memory computers and some parallel results are provided.

Parallelisation of Implicit Time Domain Methods: Progress with ADI-FDTD

The parallelisation of implicit time domain solvers is essential for applications requiring large, highly over-sampled meshes that exceed the memory limitations of standalone workstations (1). Prime application examples include large arrays of small antennas (2), and on- chip interconnects for state of the art integrated circuits (IC) (3). The need for highly over sampled meshes in these applications can be illustrated by example. Current on-chip IC interconnect widths can be as low as 22nm or less, yet cover an area > 100(mm) 2 and transmit signals with fundamental frequencies in the range DC-10GHz. Typically, transient efiects involving high frequencies are of the most interest, but even this can still require meshes with ¢x < (‚0=10 6 ), i.e., four to flve orders of magnitude smaller than for typical FDTD meshes (neglecting considerations relating to material properties). It is not practical to use explicit FDTD solvers for such meshes, even in parallel, because the Courant-Friedrichs-Lewy (CFL) stability criteria enforces a commensurate reduction in the time step size for numerical reasons. While implicit FDTD solvers, such as ADI-FDTD, ofier freedom from the CFL stability criteria, it has been presumed that parallel implementations on all but the most specialised architectures (4) would be of little beneflt due to the high communication overhead. However, we have been able to show that this is not the case (5). In this Paper we will present an overview of our work to date on the parallelisation of implicit time domain methods, including parallel ADI-FDTD (1,5) and parallel ADI-BOR-FDTD (6) on both symmetric multiprocessor (SMP) and distributed memory computer clusters (DMCC). We do not parallelise the tri-diagonal matrix solver itself, because each 2D matrix must have more than 4,000 (40,000) elements per direction before this becomes e-cient for SMP (DMCC) ma- chines. This requires 3D domains at are at or beyond current memory limits for state of the art machines. Instead, we employ domain decomposition and solve multiple, smaller, tri-diagonal ma- trix systems in parallel. Carefully organised data exchanges avoid unnecessary double-handling of data during communication, improving the parallel algorithm performance. We will describe our domain decomposition scheme, and show results for small and large domains, comparing par- allel FDTD and parallel ADI-FDTD and parallel BOR-FDTD with parallel ADI-BOR-FDTD. We demonstrate e-cient solutions of large full 3D meshes with 8 billion mesh cells for paral- lel ADI-FDTD. Since machines with SMP and DMCC architectures are widely available, our demonstration of parallel speed up represents an important step forward for the application of implicit time domain solvers for large, highly oversampled meshes with ¢x < 10 i2 ‚. We expect that our parallelisation approach can be adopted for related implicit FDTD methods.

Fast and optimal multiframe blind deconvolution algorithm for high-resolution ground-based imaging of space objects

We report a multiframe blind deconvolution algorithm that we have developed for imaging through the atmosphere. The algorithm has been parallelized to a significant degree for execution on high-performance computers, with an emphasis on distributed-memory systems so that it can be hosted on commodity clusters. As a result, image restorations can be obtained in seconds to minutes. We have compared and quantified the quality of its image restorations relative to the associated Cramér-Rao lower bounds (when they can be calculated). We describe the algorithm and its parallelization in detail, demonstrate the scalability of its parallelization across distributed-memory computer nodes, discuss the results of comparing sample variances of its output to the associated Cramér-Rao lower bounds, and present image restorations obtained by using data collected with ground-based telescopes.

An Asynchronous Parallel MLFMA for Scattering at Multiple Dielectric Objects

In this paper, a new strategy for the parallelization of the multilevel fast multipole algorithm (MLFMA) on distributed memory computers is presented. By using an asynchronous implementation of the parallel MLFMA, an efficient parallelization scheme is obtained when multiple dielectric objects are involved in the simulation. Furthermore, a better spreading of the communication through time is obtained, avoiding both communication in bursts and synchronization at each MLFMA level. This proves especially beneficial when slower interconnection networks are used.

Algorithm 881

In this article we describe our implementations of the FGMRES algorithm for both real and complex, single and double precision arithmetics suitable for serial, shared-memory, and distributed-memory computers. For the sake of portability, simplicity, flexibility, and efficiency, the FGMRES solvers have been implemented in Fortran 77 using the reverse communication mechanism for the matrix-vector product, the preconditioning, and the dot-product computations. For distributed-memory computation, several orthogonalization procedures have been implemented to reduce the cost of the dot-product calculation, which is a well-known bottleneck of efficiency for Krylov methods. Furthermore, either implicit or explicit calculation of the residual at restart is possible depending on the actual cost of the matrix-vector product. Finally, the implemented stopping criterion is based on a normwise backward error.

Parallelization of a Fast Multipole Boundary Element Method with Cluster OpenMP

Parallelization of a boundary element method for the solution of problems, which are based on a Laplace equation, is considered. The fast multipole method is applied to compress the belonging linear system of equations. The well-known parallelization standard OpenMP is used on shared memory computers and the new standard Cluster OpenMP is used on computer clusters. Both standards are based on multithreading and exploit multicore processors very efficiently. Cluster OpenMP is an enhancement of OpenMP. There, multiprocessing on a computer cluster is hidden by virtual threads, which use a virtual shared memory on a distributed memory computer.

A Sophisticated Parallel MLFMA for Scattering by Extremely Large Targets [EM Programmer's Notebook

The development of the MLFMA for realistic targets is currently at the stage where clever implementation, using a series of tricks based on the characteristics of the MLFMA and current computer technology, is required. A sophisticated parallel MLFMA for use on distributed-memory computers is presented as a whole picture of developing the program using a style of multilevel development in this paper. A series of implementation tricks for the parallel MLFMA are analyzed and compared. Particularly, a novel trick for reducing the truncation numbers is presented for extremely large targets, in the paper. These tricks are integrated into a sophisticated parallel MLFMA. The memory requirement and the CPU time for each part in each level are analyzed numerically by typical numerical experiments. The capability of this sophisticated parallel MLFMA is demonstrated by computing scattering by a sphere with a diameter of 480 wavelengths, containing around 130 million unknowns, and for a plane model with a fuselage of more than 1000 wavelengths, containing more than 72 million unknowns. These are the largest scattering problems ever solved by full-wave numerical methods, to our knowledge.

Parallel computation of three-dimensional flows using overlapping grids with adaptive mesh refinement

This paper describes an approach for the numerical solution of time-dependent partial differential equations in complex three-dimensional domains. The domains are represented by overlapping structured grids, and block-structured adaptive mesh refinement (AMR) is employed to locally increase the grid resolution. In addition, the numerical method is implemented on parallel distributed-memory computers using a domain-decomposition approach. The implementation is flexible so that each base grid within the overlapping grid structure and its associated refinement grids can be independently partitioned over a chosen set of processors. A modified bin-packing algorithm is used to specify the partition for each grid so that the computational work is evenly distributed amongst the processors. All components of the AMR algorithm such as error estimation, regridding, and interpolation are performed in parallel. The parallel time-stepping algorithm is illustrated for initial-boundary-value problems involving a linear advection–diffusion equation and the (nonlinear) reactive Euler equations. Numerical results are presented for both equations to demonstrate the accuracy and correctness of the parallel approach. Exact solutions of the advection–diffusion equation are constructed, and these are used to check the corresponding numerical solutions for a variety of tests involving different overlapping grids, different numbers of refinement levels and refinement ratios, and different numbers of processors. The problem of planar shock diffraction by a sphere is considered as an illustration of the numerical approach for the Euler equations, and a problem involving the initiation of a detonation from a hot spot in a T-shaped pipe is considered to demonstrate the numerical approach for the reactive case. For both problems, the accuracy of the numerical solutions is assessed quantitatively through an estimation of the errors from a grid convergence study. The parallel performance of the approach is examined for the shock diffraction problem.

A Parallel Multilevel Technique for Solving the Bidomain Equation on a Human Heart with Purkinje Fibers and a Torso Model

In this paper, we present a multigrid method and its implementation on parallel computers to solve the bidomain equation that appears in excitation propagation analysis of the human heart with the torso. The bidomain equation is discretized with the finite element method on a composite mesh composed of a fine voxel mesh around the heart and a coarse voxel mesh covering the torso. The extracellular potential problem on the torso is formulated as a variational problem with a constraint at the interface of the fine and coarse meshes. We show that this formulation naturally satisfies the conservation property of the electric currents and fits into the multilevel adaptive solution technique framework. We also present our special treatment of the Purkinje fiber network in the multigrid algorithm where it is modeled as multiway branching lines connected to the nodes in the voxel mesh of the heart. A parallel implementation of the proposed multigrid algorithm on distributed memory computers is presented and the performance is evaluated using real-life applications.

Time Dependent Processing in a Parallel Pipeline Architecture

Pipeline architectures provide a versatile and efficient mechanism for constructing visualizations, and they have been implemented in numerous libraries and applications over the past two decades. In addition to allowing developers and users to freely combine algorithms, visualization pipelines have proven to work well when streaming data and scale well on parallel distributed-memory computers. However, current pipeline visualization frameworks have a critical flaw: they are unable to manage time varying data. As data flows through the pipeline, each algorithm has access to only a single snapshot in time of the data. This prevents the implementation of algorithms that do any temporal processing such as particle tracing; plotting over time; or interpolation, fitting, or smoothing of time series data. As data acquisition technology improves, as simulation time-integration techniques become more complex, and as simulations save less frequently and regularly, the ability to analyze the time-behavior of data becomes more important. This paper describes a modification to the traditional pipeline architecture that allows it to accommodate temporal algorithms. Furthermore, the architecture allows temporal algorithms to be used in conjunction with algorithms expecting a single time snapshot, thus simplifying software design and allowing adoption into existing pipeline frameworks. Our architecture also continues to work well in parallel distributed-memory environments. We demonstrate our architecture by modifying the popular VTK framework and exposing the functionality to the ParaView application. We use this framework to apply time-dependent algorithms on large data with a parallel cluster computer and thereby exercise a functionality that previously did not exist.

Collective communication: theory, practice, and experience

We discuss the design and high-performance implementation of collective communications operations on distributed-memory computer architectures. Using a combination of known techniques (many of whic...

A framework for scalable greedy coloring on distributed-memory parallel computers

We present a scalable framework for parallelizing greedy graph coloring algorithms on distributed-memory computers. The framework unifies several existing algorithms and blends a variety of techniques for creating or facilitating concurrency. The latter techniques include exploiting features of the initial data distribution, the use of speculative coloring and randomization, and a BSP-style organization of computation and communication. We experimentally study the performance of several specialized algorithms designed using the framework and implemented using MPI. The experiments are conducted on two different platforms and the test cases include large-size synthetic graphs as well as real graphs drawn from various application areas. Computational results show that implementations that yield good speedup while at the same time using about the same number of colors as a sequential greedy algorithm can be achieved by setting parameters of the framework in accordance with the size and structure of the graph being colored. Our implementation is freely available as part of the Zoltan parallel data management and load-balancing library.

Scalable parallel suffix array construction

Suffix arrays are a simple and powerful data structure for text processing that can be used for full text indexes, data compression, and many other applications in particular in bioinformatics. We describe the first implementation and experimental evaluation of a scalable parallel algorithm for suffix array construction. The implementation works on distributed memory computers using MPI, Experiments with up to 512 processors show good constant factors and make it look likely that the algorithm could also be adapted to even larger systems. This makes it possible to build suffix arrays for huge inputs very quickly. Our algorithm is a parallelization of the linear time DC3 algorithm.

Parallel implementation of a semidefinite programming solver based on CSDP on a distributed memory cluster

In this paper, we present the algorithmic framework and practical aspects of implementing a parallel version of a primal–dual semidefinite programming solver on a distributed memory computer cluster. Our implementation is based on the CSDP solver and uses a message passing interface and the ScaLAPACK library. A new feature is implemented to deal with problems that have rank-one constraint matrices. We show that significant improvement is obtained for a test set of problems with rank-one constraint matrices. Moreover, we show that very good parallel efficiency is obtained for large-scale problems where the number of linear equality constraints is very large compared to the block sizes of the positive semidefinite matrix variables.

Correcting mesh-based force calculations to conserve both energy and momentum in molecular dynamics simulations

Correcting mesh-based force calculations to conserve both energy and momentum in molecular dynamics simulations