Distributed Memory Parallel Computers Research Articles

Condition number estimation of preconditioned matrices.

The present paper introduces a condition number estimation method for preconditioned matrices. The newly developed method provides reasonable results, while the conventional method which is based on the Lanczos connection gives meaningless results. The Lanczos connection based method provides the condition numbers of coefficient matrices of systems of linear equations with information obtained through the preconditioned conjugate gradient method. Estimating the condition number of preconditioned matrices is sometimes important when describing the effectiveness of new preconditionerers or selecting adequate preconditioners. Operating a preconditioner on a coefficient matrix is the simplest method of estimation. However, this is not possible for large-scale computing, especially if computation is performed on distributed memory parallel computers. This is because, the preconditioned matrices become dense, even if the original matrices are sparse. Although the Lanczos connection method can be used to calculate the condition number of preconditioned matrices, it is not considered to be applicable to large-scale problems because of its weakness with respect to numerical errors. Therefore, we have developed a robust and parallelizable method based on Hager’s method. The feasibility studies are curried out for the diagonal scaling preconditioner and the SSOR preconditioner with a diagonal matrix, a tri-daigonal matrix and Pei’s matrix. As a result, the Lanczos connection method contains around 10% error in the results even with a simple problem. On the other hand, the new method contains negligible errors. In addition, the newly developed method returns reasonable solutions when the Lanczos connection method fails with Pei’s matrix, and matrices generated with the finite element method.

Demonstration of a fully-coupled end-to-end model for small pelagic fish using sardine and anchovy in the California Current

We describe and document an end-to-end model of anchovy and sardine population dynamics in the California Current as a proof of principle that such coupled models can be developed and implemented. The end-to-end model is 3-dimensional, time-varying, and multispecies, and consists of four coupled submodels: hydrodynamics, Eulerian nutrient–phytoplankton–zooplankton (NPZ), an individual-based full life cycle anchovy and sardine submodel, and an agent-based fishing fleet submodel. A predator roughly mimicking albacore was included as individuals that consumed anchovy and sardine. All submodels were coded within the ROMS open-source community model, and used the same resolution spatial grid and were all solved simultaneously to allow for possible feedbacks among the submodels. We used a super-individual approach and solved the coupled models on a distributed memory parallel computer, both of which created challenging but resolvable bookkeeping challenges. The anchovy and sardine growth, mortality, reproduction, and movement, and the fishing fleet submodel, were each calibrated using simplified grids before being inserted into the full end-to-end model. An historical simulation of 1959–2008 was performed, and the latter 45years analyzed. Sea surface height (SSH) and sea surface temperature (SST) for the historical simulation showed strong horizontal gradients and multi-year scale temporal oscillations related to various climate indices (PDO, NPGO), and both showed responses to ENSO variability. Simulated total phytoplankton was lower during strong El Nino events and higher for the strong 1999 La Nina event. The three zooplankton groups generally corresponded to the spatial and temporal variation in simulated total phytoplankton. Simulated biomasses of anchovy and sardine were within the historical range of observed biomasses but predicted biomasses showed much less inter-annual variation. Anomalies of annual biomasses of anchovy and sardine showed a switch in the mid-1990s from anchovy to sardine dominance. Simulated averaged weights- and lengths-at-age did not vary much across decades, and movement patterns showed anchovy located close to the coast while sardine were more dispersed and farther offshore. Albacore predation on anchovy and sardine was concentrated near the coast in two pockets near the Monterey Bay area and equatorward of Cape Mendocino. Predation mortality from fishing boats was concentrated where sardine age-1 and older individuals were located close to one of the five ports. We demonstrated that it is feasible to perform multi-decadal simulations of a fully-coupled end-to-end model, and that this can be done for a model that follows individual fish and boats on the same 3-dimensional grid as the hydrodynamics. Our focus here was on proof of principle and our results showed that we solved the major technical, bookkeeping, and computational issues. We discuss the next steps to increase computational speed and to include important biological differences between anchovy and sardine. In a companion paper (Fiechter et al., 2015), we further analyze the historical simulation in the context of the various hypotheses that have been proposed to explain the sardine and anchovy cycles.

A Diskless Checkpointing Algorithm for Cluster Architectures Applied to Geospatial Raster Data Processing

In recent years, due to the increasing calculation demands for the massive spatial data analysis, the parallel computing based on high-performance computers has become an inevitable trend of geospatial raster data processing, such as digital terrain analysis (DTA for short), remote sensing interpretation and digital soil mapping. A key problem is how to design a fault-tolerant software to enhance the stability and robustness of scientific application. This paper presents an approach of failure recovery for distributed memory parallel computing. Furthermore, we adopt the master/slave programming model and present a framework of redundant master mode, which the failure occurring on the master node could not lead to a breakdown of the whole system. This approach schedules the failing task by dividing all the failing data into several partitions according to the calculating scale of failure. By means of the Fault-Tolerant Granularity Model, the scheduling algorithm can assign the failing task dynamically. Finally, taking example of digital terrain analysis, two experiments are discussed that based on the data size and the number of failures. Simulation results indicate that the proposed scheduling algorithm based on Fault-Tolerant Granularity Model achieves lower fault tolerance overhead than the rollback recovery scheme when several processors fail simultaneously.

Using distributed memory parallel computers and GPU clusters for multidimensional Monte Carlo integration

SummaryThe aim of this paper is to show that the multidimensional Monte Carlo integration can be efficiently implemented on various distributed memory parallel computers and clusters of multicore nodes using recently developed parallel versions of linear congruential generator and lagged Fibonacci generator pseudorandom number generators. We show how to accelerate the overall performance by offloading some computations to Graphics Processing Units (GPUs), and we discuss how to transform Message Passing Interface (MPI) + OpenMP programs to MPI + OpenMP + CUDA model. We explain how to utilize multiple cores of CPUs together with multiple GPU accelerators within a single node and how to achieve reasonable load balancing of all computational resources of GPU‐accelerated multicore nodes. We present and discuss the results of experiments performed on the following target architectures: IBM Blue Gene/Q parallel computer, a cluster of Intel Xeon E5‐2660 servers, and a Tesla‐based GPU cluster with Intel Xeon X5650 multicore processors. The results are presented from two points of view: strong scaling and weak scaling. We also compare the performance of all considered architectures. Copyright © 2014 John Wiley & Sons, Ltd.

Exploring the Spectrum of Dynamic Scheduling Algorithms for Scalable Distributed-MemoryRay Tracing.

This paper extends and evaluates a family of dynamic ray scheduling algorithms that can be performed in-situ on large distributed memory parallel computers. The key idea is to consider both ray state and data accesses when scheduling ray computations. We compare three instances of this family of algorithms against two traditional statically scheduled schemes. We show that our dynamic scheduling approach can render data sets that are larger than aggregate system memory and that cannot be rendered by existing statically scheduled ray tracers. For smaller problems that fit in aggregate memory but are larger than typical shared memory, our dynamic approach is competitive with the best static scheduling algorithm.

Parallel eigenvalue calculation based on multiple shift–invert Lanczos and contour integral based spectral projection method

We discuss the possibility of using multiple shift–invert Lanczos and contour integral based spectral projection method to compute a relatively large number of eigenvalues of a large sparse and symmetric matrix on distributed memory parallel computers. The key to achieving high parallel efficiency in this type of computation is to divide the spectrum into several intervals in a way that leads to optimal use of computational resources. We discuss strategies for dividing the spectrum. Our strategies make use of an eigenvalue distribution profile that can be estimated through inertial counts and cubic spline fitting. Parallel sparse direct methods are used in both approaches. We use a simple cost model that describes the cost of computing k eigenvalues within a single interval in terms of the asymptotic cost of sparse matrix factorization and triangular substitutions. Several computational experiments are performed to demonstrate the effect of different spectrum division strategies on the overall performance of both multiple shift–invert Lanczos and the contour integral based method. We also show the parallel scalability of both approaches in the strong and weak scaling sense. In addition, we compare the performance of multiple shift–invert Lanczos and the contour integral based spectral projection method on a set of problems from density functional theory (DFT).

A third-generation density-functional-theory-based method for calculating canonical molecular orbitals of large molecules

We used grid-free modified Cholesky decomposition (CD) to develop a density-functional-theory (DFT)-based method for calculating the canonical molecular orbitals (CMOs) of large molecules. Our method can be used to calculate standard CMOs, analytically compute exchange-correlation terms, and maximise the capacity of next-generation supercomputers. Cholesky vectors were first analytically downscaled using low-rank pivoted CD and CD with adaptive metric (CDAM). The obtained Cholesky vectors were distributed and stored on each computer node in a parallel computer, and the Coulomb, Fock exchange, and pure exchange-correlation terms were calculated by multiplying the Cholesky vectors without evaluating molecular integrals in self-consistent field iterations. Our method enables DFT and massively distributed memory parallel computers to be used in order to very efficiently calculate the CMOs of large molecules.

Study on Parallel Computing Method of 3-D Dynamic Analysis of Large-Scale Structure

In order to solve 3-D dynamic problem for large-scale structure, based on HDDM (Hierarchical Domain Decomposition Method) method, 3-D dynamic finite element parallel computing method of large-scale structure is studied. In this paper, BDD method and Newmark-β average acceleration method in time integration is used. Based on ADVENTRUE program, the corresponding linear implicit transient dynamic parallel program which is based on the MPI parallel environment and can be used in shared memory and distributed memory parallel computer is prepared. Several pre-conditioners are contrasted using some examples, and the results show that BDD preconditioner or BDD-DIAG preconditioner can be used in 3-D dynamic analysis for large-scale structure.

Study on Parallel Computing Method of 3-D Dynamic Analysis of Large-Scale Structure

In order to solve 3-D dynamic problem for large-scale structure, based on HDDM (Hierarchical Domain Decomposition Method) method, 3-D dynamic finite element parallel computing method of large-scale structure is studied. In this paper, BDD method and Newmark-β average acceleration method in time integration is used. Based on ADVENTRUE program, the corresponding linear implicit transient dynamic parallel program which is based on the MPI parallel environment and can be used in shared memory and distributed memory parallel computer is prepared. Several pre-conditioners are contrasted using some examples, and the results show that BDD preconditioner or BDD-DIAG preconditioner can be used in 3-D dynamic analysis for large-scale structure.

Program for quantum wave-packet dynamics with time-dependent potentials

We present a program to simulate the dynamics of a wave packet interacting with a time-dependent potential. The time-dependent Schrödinger equation is solved on a one-, two-, or three-dimensional spatial grid using the split operator method. The program can be compiled for execution either on a single processor or on a distributed-memory parallel computer. Program summaryProgram title: wavepacketCatalogue identifier: AEQW_v1_0Program summary URL:http://cpc.cs.qub.ac.uk/summaries/AEQW_v1_0.htmlProgram obtainable from: CPC Program Library, Queen’s University, Belfast, N. IrelandLicensing provisions: Standard CPC licence, http://cpc.cs.qub.ac.uk/licence/licence.htmlNo. of lines in distributed program, including test data, etc.: 7231No. of bytes in distributed program, including test data, etc.: 232209Distribution format: tar.gzProgramming language: C (iso C99).Computer: Any computer with an iso C99 compiler (e.g, gcc [1]).Operating system: Any.Has the code been vectorized or parallelized?: Yes, parallelized using MPI. Number of processors: from 1 to the number of grid points along one dimension.RAM: Strongly dependent on problem size. See text for memory estimates.Classification: 2.7.External routines: fftw [2], mpi (optional) [3]Nature of problem:Solves the time-dependent Schrödinger equation for a single particle interacting with a time-dependent potential.Solution method:The wave function is described by its value on a spatial grid and the evolution operator is approximated using the split-operator method [4, 5], with the kinetic energy operator calculated using a Fast Fourier Transform.Unusual features:Simulation can be in one, two, or three dimensions. Serial and parallel versions are compiled from the same source files.Running time:Strongly dependent on problem size. The example provided takes only a few minutes to run.

A distributed-memory parallel technique for two-dimensional mesh generation for arbitrary domains

This work describes a technique for generating two-dimensional triangular meshes using distributed memory parallel computers, based on a master/slaves model. This technique uses a coarse quadtree to decompose the domain and a serial advancing front technique to generate the mesh in each subdomain concurrently. In order to advance the front to a neighboring subdomain, each subdomain suffers a shift to a Cartesian direction, and the same advancing front approach is performed on the shifted subdomain. This shift-and-remesh procedure is repeatedly applied until no more mesh can be generated, shifting the subdomains to different directions each turn. A finer quadtree is also employed in this work to help estimate the processing load associated with each subdomain. This load estimation technique produces results that accurately represent the number of elements to be generated in each subdomain, leading to proper runtime prediction and to a well-balanced algorithm. The meshes generated with the parallel technique have the same quality as those generated serially, within acceptable limits. Although the presented approach is two-dimensional, the idea can be easily extended to three dimensions.

A large-scale nonlinear eigensolver for the analysis of dispersive nanostructures

We introduce the electromagnetic eigenmodal solver code FemaxxNano for the numerical analysis of nanometer structured optical systems, a scientific field generally know as nanooptics. FemaxxNano solves the electric field vector wave equation and calculates the electromagnetic eigenmodes of nearly arbitrary 3-dimensional resonators, embedded either in free-space, vacuum or a background medium. Here, the study of the interaction between nanometer sized metallic structures and light is at the heart of the physical problem. Since metals in the optical region of the electromagnetic spectrum are highly dispersive and, thus, dissipative, dielectric media, we eventually obtain a nonlinear eigenvalue problem. We discretize the electromagnetic eigenvalue problem with the finite element method (FEM) in 3-dimensional space and on unstructured tetrahedral grids. We introduce a fully iterative scheme to solve the nonlinear problem for complex coefficient matrices that depend on wavelength. We investigate the properties of the algorithm in detail and demonstrate its performance by analyzing a nanometer sized optical dimer structure, a specific type of optical antenna, on distributed-memory parallel computers.

Improving the Communication Pattern in Matrix-Vector Operations for Large Scale-Free Graphs by Disaggregation

Matrix-vector multiplication is the key operation in any Krylov-subspace iteration method. We are interested in Krylov methods applied to problems associated with the graph Laplacian arising from large scale-free graphs. Computations with graphs of this type on parallel distributed-memory computers are challenging. This is due to the fact that scale-free graphs have a degree distribution that follows a power law, and currently available graph partitioners are not efficient for such an irregular degree distribution. The lack of a good partitioning leads to excessive interprocessor communication requirements during every matrix-vector product. We present an approach to alleviate this problem based on embedding the original irregular graph into a more regular one by disaggregating (splitting up) vertices in the original graph. The matrix-vector operations for the original graph are performed via a factored triple matrix-vector product involving the embedding graph. Even though the latter graph is larger, we ...

PGraph: Efficient Parallel Construction of Large-Scale Protein Sequence Homology Graphs

Detecting sequence homology between protein sequences is a fundamental problem in computational molecular biology, with a pervasive application in nearly all analyses that aim to structurally and functionally characterize protein molecules. While detecting the homology between two protein sequences is relatively inexpensive, detecting pairwise homology for a large number of protein sequences can become computationally prohibitive for modern inputs, often requiring millions of CPU hours. Yet, there is currently no robust support to parallelize this kernel. In this paper, we identify the key characteristics that make this problem particularly hard to parallelize, and then propose a new parallel algorithm that is suited for detecting homology on large data sets using distributed memory parallel computers. Our method, called pGraph, is a novel hybrid between the hierarchical multiple-master/worker model and producer-consumer model, and is designed to break the irregularities imposed by alignment computation and work generation. Experimental results show that pGraph achieves linear scaling on a 2,048 processor distributed memory cluster for a wide range of inputs ranging from as small as 20,000 sequences to 2,560,000 sequences. In addition to demonstrating strong scaling, we present an extensive report on the performance of the various system components and related parametric studies.

Two-stage melting transition of bilayer systems under geometrical confinement: multicolour domain decomposition Monte Carlo simulation

The nature of the melting transition in a thin film consisting of a bilayer of spheres confined between parallel walls was investigated by the multicolour domain decomposition Monte Carlo (MDDMC) method. The MDDMC method is suitable for distributed memory parallel computers, and the order parameters of the system were calculated numerically. The nature of the melting transition from an ordered phase with square lattice symmetry was clarified, with the number of particles N larger than 105. The profile of the bond-orientational correlation function near the transition point was also shown. At an intermediate density, this function was found to obey a power-law behaviour such as . A comparison with recent experiments is also presented.

Large-scale time-harmonic electromagnetic field analysis using a multigrid solver on a distributed memory parallel computer

This paper reports on an investigation into large-scale parallel time-harmonic electromagnetic field analysis based on the finite element method. The parallel geometric multigrid preconditioned iterative solver for the resulting linear system was developed on a cluster of shared memory parallel computers. We propose a hybrid parallel ordering method for the parallelization of a multiplicative Schwarz smoother, which is a key component of the multigrid solver for electromagnetic field analysis. The method, using domain decomposition ordering for multi-process parallelism and introducing block multi-color ordering for multi-thread parallel processing, attains a high convergence rate with a small number of message passing interface communications and thread synchronizations. The numerical test confirms that the proposed method attains a solver performance more than twice as good as the conventional method based on multi-color ordering. Furthermore, an approximately 800 million degrees of freedom problem is successfully solved on 256 quad-core processors.

UPC-CHECK: a scalable tool for detecting run-time errors in Unified Parallel C

Unified Parallel C (UPC) is a language used to write parallel programs for distributed memory parallel computers. UPC-CHECK ( http://hpcgroup.public.iastate.edu/UPC-CHECK/ ) is a scalable tool developed to automatically detect argument errors in UPC functions and deadlocks in UPC programs at run-time and issue high quality error messages to help programmers quickly fix those errors. The run-time complexity of all detection techniques used are optimal, i.e. O(1) except for deadlocks involving locks where it is theoretically known to be linear in the number of threads. The tool is easy to use, and involves merely replacing the compiler command with upc-check. Error messages issued by UPC-CHECK were evaluated using the UPC RTED test suite for argument errors in UPC functions and deadlocks. Results of these tests show that the error messages issued by UPC-CHECK for these tests are excellent.

A Parallel Algorithm to Calculate the Costrank of a Network

We developed analogous parallel algorithms to implement CostRank for distributed memory parallel computers using multi processors. Our intent is to make CostRank calculations for the growing number of hosts in a fast and a scalable way. In the same way we intent to secure large scale networks that require fast and reliable computing to calculate the ranking of enormous graphs with thousands of vertices (states) and millions or arcs (links). In our proposed approach we focus on a parallel CostRank computational architecture on a cluster of PCs networked via Gigabit Ethernet LAN to evaluate the performance and scalability of our implementation. In particular, a partitioning of input data, graph files, and ranking vectors with load balancing technique can improve the runtime and scalability of large-scale parallel computations. An application case study of analogous Cost Rank computation is presented. Applying parallel environment models for one-dimensional sparse matrix partitioning on a modified research page, results in a significant reduction in communication overhead and in per-iteration runtime.We provide an analytical discussion of analogous algorithms performance in terms of I/O and synchronization cost, as well as of memory usage. General Terms Security, Algorithms, Networking.

503 非構造格子を用いた分散メモリ型並列計算に関する研究(GS-5 CFD)

503 非構造格子を用いた分散メモリ型並列計算に関する研究(GS-5 CFD)

1403 10億粒子解析のためのマルチスケールグリッドを用いた階層型領域分割MPS陽解法と津波解析への応用(OS14-1.メッシュフリー/粒子法とその関連技術(1),OS・一般セッション講演)

In this research, a new distributed memory parallel algorithm of the explicit MPS (Moving Particle Simulation) method is presented. An analysis region is divided for a distributed memory parallel computation using ParMETIS. As a result, we have been able to do the MPS analysis of 523 million particles using 23,040 cores in 5.4 seconds per one time step. In the performance, we has achieved near 5% of the peak performance, 16.65 teraflops at maximum and 13.24 speed-up from 480 cores to 7,680 cores. Performance during large scale simulation is examined by computing tsunami wave run-up on a virtual gulf area using up to 58 million particles.