IBM SP2 Research Articles

A Hessian-free Newton–Raphson method for the configuration of physics systems featured by numerically asymmetric force field

Numerically asymmetric force field is examined in particle-oriented problems such as Quasicontinuum modeling and simulation. To configure the ground state of a large-scale physical system featured by numerically asymmetric force field, we propose a Hessian-free Newton–Raphson method where the Newton equation is solved using central-difference based BiCGstab algorithm (denoted as HFNR-BiCGstab-diff for simplicity). A detailed analytical and experimental investigation on the convergence performance of the HFNR-BiCGstab-diff algorithm is given in this paper. A pure HFNR-BiCGstab-diff algorithm may suffer from unreliable start-up, particularly in the case that the initial guess is far from the minimizer. As a remedy, a hybrid method that couples HFNR-BiCGstab-diff with preconditioned nonlinear conjugate gradient algorithm (PNCG) is developed to achieve optimal computational performance. The algorithms addressed in this paper have been implemented using POSIX-Thread C++. Their performance has been evaluated using three-dimensional Quasicontinuum simulation problems, which are featured by asymmetric force field and large dimensional sizes up to 122,808 degree-of-freedom, as benchmarks. The numerical experiment on IBM SP2 demonstrates that, compared to alternative unconstrained optimization methods such as preconditioned nonlinear conjugate gradient algorithm, the hybrid algorithm saves 20%–60% running time.

Towards a three-dimensional moving body incompressible flow solver with a linear deformable model

In this study, a three-dimensional fluid–structured parallelized solver is extended from the previous work (Niu et al., 2009 [1]) for moving body simulations. Based on the unified Eulerian and Lagrangian coordinate transformations, the unsteady three-dimensional incompressible Navier–Stokes equations with artificial compressibility (Chorin, 1967 [2]) in a dual-time stepping approach are first derived. To implement unsteady flow calculations, the dual-time stepping strategy including the LU decomposition method is used in the pseudo-time iteration and the second-order accurate backward difference is adopted to discretize the unsteady flow terms. Also, a third-order Roe type flux limited splitting is derived to evaluate the spatial difference of the convective fluxes. The original FORTRAN code is converted to the MPI code and tested on a 64-CPU IBM SP2. The parallel strategy here is based on the partitions of all do-loops in the original FORTRAN code and transferring the calculations inside the do-loop into different CPUs. The partition of the do-loop can be applied on the innermost loop, only or the last two inner loops depending on two-dimensional or three-dimensional problems. This kind of the parallel data partition of the loops is independent of what kind of the explicit or implicit type numerical algorithm used. Therefore, the current parallel approach can take advantage of the MPI language fully to transfer data efficiently among CPUs even for solving the governing equation implicitly. The test results show that a significant reduction of computing time in running the model and a near-linear speed up rate is achieved up to 32 CPUs at IBM SP2. The speed up rate is as high as 31 for using 64 IBM SP2 processors The test shows efficient parallel processing to provide prompt simulation of 3D cavity, unsteady dropping airfoil and blood flows in an aortic tube with a linear elastic modeling of wall motion is included here.

Simulation of flowfields induced by wind blades based on a parallelized low-speed flow solver

In this study, a parallel computing technology is applied on the simulation of a wind turbine flow problem. A third-order Roe type flux limited splitting based on a pre-conditioning matrix with an explicit time marching method is used to solve the Navier–Stokes equations. The original FORTRAN code was parallelized with Message Passing Interface (MPI) language and tested on a 64-CPU IBM SP2 parallel computer. The test results show that a significant reduction of computing time in running the model and a super-linear speed up rate is achieved up to 32 CPUs at IBM SP2 processors. The speed up rate is as high as 49 for using IBM SP2 64 processors. The test shows very promising potential of parallel processing to provide prompt simulation of the current wind turbine problems.

A NUMERICAL SIMULATION OF CARDIAC ELECTRIC ACTIVITY IN LV BASED ON MONO-DOMAIN MODEL

Heterogeneities of ionic current expression and electrophysiological characteristics exist within the mammalian ventricle wall and are assumed to be the result of regional differences in myocardial membrane ionic current densities. This work describes computational approaches of modeling the cardiac cellular excitability and electrical activity propagation in the spatially inhomogeneous ventricular tissue. A three-dimensional (3D) unsteady Rogers model with one gating variable has been considered for simulating the cardiac electrical activity. Based on the paradigm of domain decomposition a 3D finite element scheme has been developed and implemented using PETSc Library under MPI environment. The numerical simulations are carried out on IBM SP3 machine. The computational studies provide insight into the underlying mechanisms during the normal and pathological cardiac excitation.

On a parallel genetic–tabu search based algorithm for solving the graph colouring problem

In this paper, we are interested in a particular combinatorial optimisation problem (COP), namely the graph colouring problem (GCP). To solve the GCP, we present a parallel approach adopting an efficient strategy. A brief survey on known methods for solving the GCP enables us to justify our approach which is based on a hybrid method, starting from a set of solutions initialized by the so-called RLF colouring method and combining both a genetic algorithm and the tabu search. A parallelising strategy is then applied. The performances of our method were evaluated through a series of experimentations achieved on an IBM SP2 multiprocessor. The processed graphs were chosen from two benchmark sets. The first, taken from the Internet, involves graphs whose chromatic numbers are known and the second involves random generated graphs. The analysis of the results proves the interest of our approach.

SPARSE APPROXIMATE INVERSE PRECONDITIONER FOR CONTACT PROBLEMS ON THE EARTH SIMULATOR USING OPENMP

In this study, the sparse approximate inverse (SAI) preconditioning method is applied to a single SMP node of the Earth Simulator (ES) using OpenMP. The developed method is tested for applications with contact conditions, and demonstrates efficiency and robustness for a wide range of problem sizes. The efficiency and robustness of the SAI preconditioner is competitive with that of the selective blocking method, which is a special preconditioning method for contact problems. The developed method is ported to Hitachi SR8000 and IBM SP-3, and the performance and robustness is evaluated.

Efficient Parallel I/O in Community Atmosphere Model (CAM)

Century-long global climate simulations at high resolutions generate large amounts of data in a parallel architecture. Currently, the community atmosphere model (CAM), the atmospheric component of the NCAR community climate system model (CCSM), uses sequential I/O which causes a serious bottleneck for these simulations. We describe the parallel I/O development of CAM in this paper. The parallel I/ O combines a novel remapping of 3-D arrays with the parallel netCDF library as the I/O interface. Because CAM history variables are stored in disk file in a different index order than the one in CPU resident memory because of parallel decomposition, an index reshuffle is done on the fly. Our strategy is first to remap 3-D arrays from its native decomposition to z-decomposition on a distributed architecture, and from there write data out to disk. Because z-decomposition is consistent with the last array dimension, the data transfer can occur at maximum block sizes and, therefore, achieve maximum I/ O bandwidth. We also incorporate the recently developed parallel netCDF library at Argonne/Northwestern as the collective I/O interface, which resolves a long-standing issue because netCDF data format is extensively used in climate system models. Benchmark tests are performed on several platforms using different resolutions. We test the performance of our new parallel I/O on five platforms (SP3, SP4, SP5, Cray X1E, BlueGene/L) up to 1024 processors. More than four realistic model resolutions are examined, e.g. EUL T85 (~1.4°), FV-B (2° × 2.5°), FV-C (1° × 1.25°), and FV-D (0.5° × 0.625°) resolutions. For a standard single history output of CAM 3.1 FV-D resolution run (multiple 2-D and 3-D arrays with total size 4.1 GB), our parallel I/O speeds up by a factor of 14 on IBM SP3, compared with the existing I/O; on IBM SP5, we achieve a factor of 9 speedup. The estimated time for a typical century-long simulation of FV D-resolution on IBM SP5 shows that the I/O time can be reduced from more than 8 days (wall clock) to less than 1 day for daily output. This parallel I/O is also implemented on IBM BlueGene/ L and the results are shown, whereas the existing sequential I/O fails due to memory usage limitation.

Parallelization methods for implementation of a magnetic induction tomography forward model in symmetric multiprocessor systems

This paper describes four parallelization approaches used in a finite-difference-based electromagnetic modeller for application in magnetic induction tomography (MIT) and suitable for implementation on computer systems with symmetric multiprocessor (SMP) architecture. The approaches include: (i) splitting by coils using a distributed memory approach, (ii) splitting by physical domain using a distributed memory approach, (iii) splitting by physical domain using hybrid distributed/shared memory approach and (iv) splitting by both coils and physical domain using multi-level distributed and shared memory approaches respectively. All four approaches were implemented and tested on an IBM SP supercomputer. Coil parallelization was the most efficient method due to low inter-processor communication requirements but was limited by the number of coils in the MIT system. Approaches (ii) and (iii) allowed a larger number of processors to be employed but the efficiency versus number of processors was found to drop at a faster rate in comparison to (i). The fourth approach both allowed a larger number of processors to be employed and was found to provide higher efficiency than the parallelization by physical domain only. This multi-level hybrid approach therefore appears to offer an effective parallelization method for implementation of the MIT forward model on SMP clusters.

Scalability of Neville elimination using checkerboard partitioning

The scalability of a parallel system is a measure of its capacity to effectively use an increasing number of processors. Both the isoefficiency function and the scaled efficiency are metrics used to analyse the scalability of parallel algorithms and architectures. The first function relates the size of the problem being solved to the number of processors required to maintain efficiency at a fixed value, while the second function shows how an algorithm scales when both the size of the problem and the number of processors are increased. This paper models and measures the parallel scalability of the Neville method when a checkerboard partitioning is performed. Neville elimination is a method for solving a system of linear equations. This process appears naturally when the Neville strategy of interpolation is used to solve linear systems. The scaled efficiency of some algorithms of this method is studied on an IBM SP2 and also over an HP cluster.

Solution of Linear Systems by GMRES Method on Global Computing Platform

By making use of a very large amount of unexploited computing resources, grid computing achieves high throughput computing. We present a classical parallel method GMRES (m) to solve large sparse linear systems utilizing a lightweight GRID system XtremWeb. XtremWeb is a global computing platform which is dedicated to multi-parameters generic applications. We have implemented this important algorithm GMRES (m) which is one of the key methods to resolve large, non-symmetric, linear problems on this system. We discuss as well the performances of this implementation deployed on two XtremWeb networks: a local network with 128 non-dedicated PCs in Polytech-Lille of University of Sciences and Technologies of Lille, France, a remote network with 3 clusters of SCGN Grid including 91 CPUs totally in the High Performance Computing Center of University of Tsukuba in Japan. We also performed the tests on the platform of supercomputer IBM SP4 of CINES in Montpellier, France. We compare the performances on the two different computing systems. The advantages and drawbacks of our implementations on this GRID computing system XtremWeb will be well explained.

Hybrid Runtime Management of Space-Time Heterogeneity for Parallel Structured Adaptive Applications

Structured adaptive mesh refinement (SAMR) techniques provide an effective means for dynamically concentrating computational effort and resources to appropriate regions in the application domain. However, due to their dynamism and space-time heterogeneity, scalable parallel implementation of SAMR applications remains a challenge. This paper investigates hybrid runtime management strategies and presents an adaptive hierarchical multipartitioner (AHMP) framework. AHMP dynamically applies multiple partitioners to different regions of the domain, in a hierarchical manner, to match the local requirements of the regions. Key components of the AHMP framework include a segmentation-based clustering algorithm (SBC) that can efficiently identify regions in the domain with relatively homogeneous partitioning requirements, mechanisms for characterizing the partitioning requirements of these regions, and a runtime system for selecting, configuring, and applying the most appropriate partitioner to each region. Further, to address dynamic resource situations for long-running applications, AHMP provides a hybrid partitioning strategy (HPS) that involves application-level pipelining, trading space for time when resources are sufficiently large and underutilized, and an application-level out-of-core strategy (ALOC), trading time for space when resources are scarce in order to enhance the survivability of applications. The AHMP framework has been implemented and experimentally evaluated on up to 1,280 processors of the IBM SP4 cluster at the San Diego Supercomputer Center.

Data distribution schemes of sparse arrays on distributed memory multicomputers

A data distribution scheme of sparse arrays on a distributed memory multicomputer, in general, is composed of three phases, data partition, data distribution, and data compression. To implement the data distribution scheme, many methods proposed in the literature first perform the data partition phase, then the data distribution phase, followed by the data compression phase. We called a data distribution scheme with this order as Send Followed Compress (SFC) scheme. In this paper, we propose two other data distribution schemes, Compress Followed Send (CFS) and Encoding-Decoding (ED), for sparse array distribution. In the CFS scheme, the data compression phase is performed before the data distribution phase. In the ED scheme, the data compression phase can be divided into two steps, encoding and decoding. The encoding step and the decoding step are performed before and after the data distribution phase, respectively. To evaluate the CFS and the ED schemes, we compare them with the SFC scheme. In the data partition phase, the row partition, the column partition, and the 2D mesh partition with/without load-balancing methods are used for these three schemes. In the compression phase, the CRS/CCS methods are used to compress sparse local arrays for the SFC and the CFS schemes while the encoding/decoding step is used for the ED scheme. Both theoretical analysis and experimental tests were conducted. In the theoretical analysis, we analyze the SFC, the CFS, and the ED schemes in terms of the data distribution time and the data compression time. In experimental tests, we implemented these three schemes on an IBM SP2 parallel machine. From the experimental results, for most of test cases, the CFS and the ED schemes outperform the SFC scheme. For the CFS and the ED schemes, the ED scheme outperforms the CFS scheme for all test cases.

Enabling scalable parallel implementations of structured adaptive mesh refinement applications

Parallel implementations of dynamic structured adaptive mesh refinement (SAMR) methods lead to significant runtime management challenges that can limit their scalability on large systems. This paper presents a runtime engine that addresses the scalability of SAMR applications with localized refinements and high SAMR efficiencies on large numbers of processors (upto 1024 processors). The SAMR runtime engine augments hierarchical partitioning with bin-packing based load-balancing to manage the space-time heterogeneity of the SAMR grid hierarchy, and includes a communication substrate that optimizes the use of MPI non-blocking communication primitives. An experimental evaluation on the IBM SP2 supercomputer using the 3-D Richtmyer-Meshkov compressible turbulence kernel demonstrates the effectiveness of the runtime engine in improving SAMR scalability.

TRLE—an efficient data compression scheme for image composition of volume rendering on distributed memory multicomputers

Data compression is a well-known method to improve the image composition time of parallel volume rendering on distributed memory multicomputers. In this paper, we propose an efficient data compression scheme, the template run-length encoding (TRLE) scheme, for image composition. Given an image with 2n×2n pixels, in the TRLE scheme, the image is treated as n×n blocks and each block has 2×2 pixels. Since a pixel can be a blank or non-blank pixel, there 16 templates in a block. To compress an image, the TRLE scheme encodes an image block by block similar to the run-length encoding scheme. However, the TRLE scheme can filter out or use small space to encode blocks whose four pixels are blank pixels, that is, the TRLE scheme can encode a partial image according to the shape of non-blank pixels. To evaluate the performance of the TRLE scheme, we compare the proposed scheme with the BR, the RLE, and the BRLC schemes. Since a data compression scheme needs to cooperate with some data communication schemes, in the implementation, the binary-swap, the parallel-pipelined, and the rotate-tiling data communication schemes are used. By combining the four data compression schemes with the three data communication schemes, we have twelve image composition methods. These twelve methods are implemented on an IBM SP2 parallel machine. Four volume datasets are used as test samples. The data computation time and the data communication time are measured. The experimental results show that the TRLE data compression scheme with the rotate-tiling data communication scheme outperforms other eleven image composition methods for all test samples.

An MCHF atomic-structure package for large-scale calculations

An MCHF atomic-structure package for large-scale calculations

Analyzing Scalability of Neville Elimination

The scalability of a parallel system is a measure of its capacity to effectively use an increasing number of processors. Several performance evaluation metrics have been developed to study the scalability of parallel algorithms and architectures. The isoefficiency function is one of those metrics. It relates the size of the problem being solved to the number of processors required to maintain efficiency at a fixed value. This work studies the scalability of Neville elimination, which is a method to solve a linear equation system. This process appears naturally when the Neville strategy of interpolation is used to solve linear systems. The scalability behavior of some algorithms of this method is studied on an IBM SP2 and also over a network of personal computers using the isoefficiency function and the scaled efficiency.

A hybrid GMRES/LS-arnoldi method to accelerate the parallel solution of linear systems

We present a parallel hybrid asynchronous method to solve large sparse linear systems by the use of a large parallel machine. This method combines a parallel GMRES( m) algorithm with the least squares method that needs some eigenvalues obtained from a parallel Arnoldi algorithm. All of the algorithms run on different processors of an IBM SP3 or IBM SP4 computer simultaneously. This implementation of this hybrid method allows us to take advantage of the parallelism available and to accelerate the convergence by decreasing considerably the number of iterations.

Essential Cycle Calculation Method for Irregular Array Redistribution

In many parallel programs, run-time array redistribution is usually required to enhance data locality and reduce remote memory access on the distributed memory multicomputers. In general, array distribution can be classified into regular distribution and irregular distribution according to the distribution fashion. Many methods for performing regular array redistribution have been presented in the literature. However, for the heterogeneous computation environment, irregular array redistributions can be used to adjust data assignment at run-time. In this paper, an Essential Cycle Calculation method for unequal block sizes array redistribution is presented. In the ECC method, a processor first computes the source/destination processor/data sets of array elements in the first essential cycle of the local array it owns. From the source/destination processor/data sets of array elements in the first essential cycle, we can construct packing/unpacking pattern tables. Since each essential cycle has the same communication pattern, based on the packing/unpacking pattern tables, a processor can pack/unpack array elements efficiently. To evaluate the performance of the ECC method, we have implemented this method on an IBM SP2 parallel machine and compare it with the Sequence method. The cost models for these methods are also presented. The experimental results show that the ECC method greatly outperforms the Sequence method for all test samples.

A parallel approach to STAP implementation for fMRI data

To exploit the capabilities of parallel processing in applying the space-time adaptive processing (STAP) algorithm, previously explored on a small scale for functional magnetic resonance imaging (fMRI) applications, to conventional size fMRI data sets. STAP is a two-dimensional filter that is able to locate fMRI activations in both space and frequency. It is applied here for the construction of brain activation maps in fMRI using Visual Age C, incorporating Engineering and Scientific Subroutine Library (ESSL) functions, compiled in 64-bit, and executed on an IBM SP supercomputer. Computer simulations incorporating actual MRI noise indicate that STAP, incorporated using the method of steepest descent, is feasible on conventional size data sets and exhibits an improvement in detecting activations over the more traditional cross correlation method of fMRI analysis when the response is unknown. STAP is feasible on traditional size fMRI data sets and useful in elucidating spatial and temporal connectivity.

Parallel divide-and-conquer scheme for 2D Delaunay triangulation

This work describes a parallel divide-and-conquer Delaunay triangulation scheme. This algorithm finds the affected zone, which covers the triangulation and may be modified when two sub-block triangulations are merged. Finding the affected zone can reduce the amount of data required to be transmitted between processors. The time complexity of the divide-and-conquer scheme remains O(n log n), and the affected region can be located in O(n) time steps, where n denotes the number of points. The code was implemented with C, FORTRAN and MPI, making it portable to many computer systems. Experimental results on an IBM SP2 show that a parallel efficiency of 44–95% for general distributions can be attained on a 16-node distributed memory system. Copyright © 2006 John Wiley & Sons, Ltd.