Large-scale Parallel Computers Research Articles

High-fidelity coupled Monte Carlo neutron transport and thermal-hydraulic simulations using Serpent 2/SUBCHANFLOW

Efforts to develop high-fidelity, in silico or ab initio, high performance multi-physics tools are undertaken by many groups due to the availability of relatively cheap, large-scale parallel computers. To this end, an internal coupling between the Monte Carlo reactor physics code Serpent 2 and the sub-channel code SUBCHANFLOW has been developed. The coupled code system is intended to serve as reference for deterministic reactor dynamics code developments in the future. It exploits the fact that Serpent was conceived as a lattice code for such deterministic tools. The coupling utilizes Serpent’s recently introduced universal multi-physics interface. With the multi-physics interface enabled, Serpent treats temperature dependence of nuclear data using the target motion sampling method. Since the target motion sampling methodology cannot be applied to thermal bound-atom scattering or unresolved resonances, a stochastic mixing fall back algorithm to enable the simulation of thermal reactors has been implemented. The developed coupled code is verified by code-to-code comparison with an external coupling of the Monte Carlo tool TRIPOLI4 and SUBCHANFLOW as well as the internally coupled code MCNP5/SUBCHANFLOW. Simulation results of all code systems were found to be in good agreement. Thereafter, the second exercise of the OECD/NEA and U.S. NRC PWR MOX/UO2 core transient benchmark is studied to demonstrate that Serpent 2/SUBCHANFLOW may be employed to analyze realistic, industry-like cases such as a full PWR core under hot full power conditions in a reasonable amount of time. The obtained simulation results are compared to known benchmark solutions and the numerical performance of Serpent 2/SUBCHANFLOW is analyzed to assess the feasibility of routine application. While Serpent 2/SUBCHANFLOW’s performance in terms of physics and numerical efficiency is found to be generally satisfactory, options to further improve the coupled tool concerning both aspects are discussed. Afterwards, first efforts to validate Serpent 2/SUBCHANFLOW using the hot zero power state of the cycle 1 of the BEAVRS benchmark are presented.

Parallel heuristics for scalable community detection

Community detection has become a fundamental operation in numerous graph-theoretic applications. It is used to reveal natural divisions that exist within real world networks without imposing prior size or cardinality constraints on the set of communities. Despite its potential for application, there is only limited support for community detection on large-scale parallel computers, largely owing to the irregular and inherently sequential nature of the underlying heuristics. In this paper, we present parallelization heuristics for fast community detection using the Louvain method as the serial template. The Louvain method is an iterative heuristic for modularity optimization. Originally developed by Blondel et al. in 2008, the method has become increasingly popular owing to its ability to detect high modularity community partitions in a fast and memory-efficient manner. However, the method is also inherently sequential, thereby limiting its scalability. Here, we observe certain key properties of this method that present challenges for its parallelization, and consequently propose heuristics that are designed to break the sequential barrier. For evaluation purposes, we implemented our heuristics using OpenMP multithreading, and tested them over real world graphs derived from multiple application domains (e.g., internet, citation, biological). Compared to the serial Louvain implementation, our parallel implementation is able to produce community outputs with a higher modularity for most of the inputs tested, in comparable number or fewer iterations, while providing absolute speedups of up to 16x using 32 threads.

Multi-scale Coupling Simulation of Seismic Waves and Building Vibrations Using ppOpen-HPC

Abstract In order to simulate an earthquake shock originating from the earthquake source and the damage it causes to buildings, not only the seismic wave that propagates over a wide region of several 100 km2, but also the building vibrations that occur over a small region of several 10 m2 must be resolved concurrently. Such a multi-scale simulation is difficult because such kind of modeling and implementation by only a specific application are limited. To overcome these problems, a multi-scale weak-coupling simulation of seismic wave and building vibrations using “ppOpen-HPC” libraries is conducted. The ppOpen-HPC, wherein “pp” stands for “post-peta scale”, is an open source infrastructure for development and execution of optimized and reliable simulation codes on large-scale parallel computers. On the basis of our evaluation, we confirm that an acceptable result can be achieved that ensures that the overhead cost of the coupler is negligible and it can work on large-scale computational resources.

A parallel space-time domain decomposition method for unsteady source inversion problems

In this paper, we propose a parallel space-time domain decomposition method for solving an unsteady source identification problem governed by the linear convection-diffusion equation. Traditional approaches require to solve repeatedly a forward parabolic system, an adjoint system and a system with respect to the unknowns. The three systems have to be solved one after another. These sequential steps are not desirable for large scale parallel computing. A space-time restrictive additive Schwarz method is proposed for a fully implicit space-time coupled discretization scheme to recover the time-dependent pollutant source intensity functions. We show with numerical experiments that the scheme works well with noise in the observation data. More importantly it is demonstrated that the parallel space-time Schwarz preconditioner is scalable on a supercomputer with over $10^3$ processors, thus promising for large scale applications.

Direct solution method of efficient large-scale parallel computation for 3D turbulent Rayleigh-Bénard convection

Research of turbulence Rayleigh-Bénard convection with high Ra number is a hot topic in physics research in the world. DNS simulation is one of the important means to study the subject. The computing work is hard to achieve when the calculation size is increased and the grid number is bigger. Numerical simulation for high Ra turbulent convection faces some major challenges. So the direct (non iterative) solution method of efficient large-scale parallel computation for the 3D turbulent convection is created in this paper. Main difficulties are the parallel computing technology for the pressure Poisson equation. The mass efficient parallel approximate solution with the block tridiagonal equations of OpenMP and MPI used simultaneously after decoupling pressure Poisson equation using FFT is presented. Through the validation of the efficiency of this method in parallel computing, the new method for direct solution of parallel computing have good parallel efficiency and computational time. Results of thermal convection in 3D narrow cavity show that the convection characteristics calculated by using the new method is reasonable. The direct solution method for efficient large-scale parallel computation of 3D turbulent convection created in this paper also is likely to be a breakthrough in computing technology about efficient large-scale parallel computing incompressible NS equations in some special cases.

Performance Analysis of Homogeneous On-Chip Large-Scale Parallel Computing Architectures for Data-Parallel Applications

On-chip computing platforms are evolving from single-core bus-based systems to many-core network-based systems, which are referred to asOn-chip Large-scale Parallel Computing Architectures (OLPCs)in the paper. Homogenous OLPCs feature strong regularity and scalability due to its identical cores and routers. Data-parallel applications have their parallel data subsets that are handled individually by the same program running in different cores. Therefore, data-parallel applications are able to obtain good speedup in homogenous OLPCs. The paper addresses modeling the speedup performance of homogeneous OLPCs for data-parallel applications. When establishing the speedup performance model, the network communication latency and the ways of storing data of data-parallel applications are modeled and analyzed in detail. Two abstract concepts (equivalent serial packet and equivalent serial communication) are proposed to construct the network communication latency model. The uniform and hotspot traffic models are adopted to reflect the ways of storing data. Some useful suggestions are presented during the performance model’s analysis. Finally, three data-parallel applications are performed on our cycle-accurate homogenous OLPC experimental platform to validate the analytic results and demonstrate that our study provides a feasible way to estimate and evaluate the performance of data-parallel applications onto homogenous OLPCs.

Logic circuit design for high-speed computing of dynamic response in real-time hybrid simulation using FPGA-based system

One of the issues in extending the range of applicable problems of real-time hybrid simulation is the computation speed of the simulator when large-scale computational models with a large number of DOF are used. In this study, functionality of real-time dynamic simulation of MDOF systems is achieved by creating a logic circuit that performs the step-by-step numerical time integration of the equations of motion of the system. The designed logic circuit can be implemented to an FPGA-based system; FPGA (Field Programmable Gate Array) allows large-scale parallel computing by implementing a number of arithmetic operators within the device. The operator splitting method is used as the numerical time integration scheme. The logic circuit consists of blocks of circuits that perform numerical arithmetic operations that appear in the integration scheme, including addition and multiplication of floating-point numbers, registers to store the intermediate data, and data busses connecting these elements to transmit various information including the floating-point numerical data among them. Case study on several types of linear and nonlinear MDOF system models shows that use of resource sharing in logic synthesis is crucial for effective application of FPGA to real-time dynamic simulation of structural response with time step interval of 1 ms.

Analysis of the second order accurate uniform equilibrium flux method and its graphics processing unit acceleration

The extension of the Uniform Equilibrium Flux Method (UEFM) to second order accuracy in space is presented. This extension is made possible through the recasting of the original UEFM flux expressions from a volumetric flux to a surface flux, allowing for reconstruction through a Taylor series expansion of the resulting split surface fluxes at the cell interfaces. By doing so, we avoid the difficulties associated with integration of the gradient terms over velocity and physical space as required by the original UEFM fluxes. Analysis of the dissipative qualities of the renewed direction split UEFM flux expressions demonstrate that the numerical dissipation is a function of Mach number, with increasing amounts of dissipation present with increasing Mach numbers. Following this analysis, the higher order UEFM fluxes are applied to large scale parallel computation using Graphics Processing Units, or GPUs, through the use of CUDA. The vector split nature of the UEFM fluxes are well suited to GPU computation due to the high degree of locality. This parallelization is performed using a cell-based parallel paradigm through the creation of several key CUDA kernels for the calculation of split fluxes, gradient of split fluxes and state related computations. The algorithm is executed entirely on the GPU device, with the host remaining idle during the computation stage. The GPU accelerated UEFM algorithm is then applied to the solution of several two dimensional benchmark problems. Speedup of approximately 200 and 171 times for first order accuracy and second order accuracy respectively is demonstrated when using an Nvidia Tesla C2075 computing GPU compared to that of a single core of an Intel Xeon E5-2760 CPU.

Micromagnetic Simulations of Magnetization Reversal in Misaligned Multigrain Magnets With Various Grain Boundary Properties Using Large-Scale Parallel Computing

This paper reports on micromagnetic simulations of the magnetization reversal behavior in polycrystalline Nd-Fe-B sintered magnets using large-scale parallel computing. A multigrain model is introduced to calculate the grain alignment dependence of the coercivity of polycrystalline Nd-Fe-B magnets with various magnetic characteristics at grain boundaries (GBs). The magnetic domain wall motion is accurately treated by dividing the analyzed object into extremely small elements. The multigrain model with a soft magnetic GB phase and a reverse domain at the initial state well reproduces experimental results. The calculations of coercivity with several GB widths are also carried out to seek for the origin of the sudden decrease of coercivity with nearly perfect grain alignment.

Toward large scale parallel computer simulation of viscoelastic fluid flow: A study of benchmark flow problems

Followed by our previous study, an OpenFOAM-based viscoelastic flow solver has been further validated through simulation of viscoelastic flow past a cylinder. The drag coefficients calculated by the Oldroyd-B model under the creeping flow in a range of Weissenberg (Wi) number are in good agreements with those reported in the literature. Using the linear Phan-Thien Tanner (L-PTT) model, time-dependent two-dimensional simulations of flow past cylinder have been carried out in a range of Wi number and Reynolds (Re) number, and revealed interesting cooperative effects of inertia and elasticity on the structural evolution of the wake behind the cylinder. The details of parallel computing strategy are analysed and discussed. The codes are evaluated for large scale parallel simulation of two-dimensional and three-dimensional contraction flow as well as two-dimensional flow past a cylinder. The key bottlenecks, which affect the scalability of parallel computing, are discussed.

High performance computing for flood simulation using Telemac based on hybrid MPI/OpenMP parallel programming

Usually simulations on environment flood issues will face the scalability problem of large scale parallel computing. The plain parallel technique based on pure MPI is difficult to have a good scalability due to the large number of domain partitioning. Therefore, the hybrid programming using MPI and OpenMP is introduced to deal with the issue of scalability. This kind of parallel technique can give a full play to the strengths of MPI and OpenMP. During the parallel computing, OpenMP is employed by its efficient fine grain parallel computing and MPI is used to perform the coarse grain parallel domain partitioning for data communications. Through the tests, the hybrid MPI/OpenMP parallel programming was used to renovate the finite element solvers in the BIEF library of Telemac. It was found that the hybrid programming is able to provide helps for Telemac to deal with the scalability issue.

Large-scale parallel computation of the phonon Boltzmann Transport Equation

Non-equilibrium heat conduction, as occurring in modern-day sub-micron semiconductor devices, can be predicted effectively using the Boltzmann Transport Equation (BTE) for phonons. In this article, strategies and algorithms for large-scale parallel computation of the phonon BTE are presented. An unstructured finite volume method for spatial discretization is coupled with the control angle discrete ordinates method for angular discretization. The single-time relaxation approximation is used to treat phonon–phonon scattering. Both dispersion and polarization of the phonons are accounted for. Three different parallelization strategies are explored: (a) band-based, (b) direction-based, and (c) hybrid band/cell-based. Subsequent to validation studies in which silicon thin-film thermal conductivity was successfully predicted, transient simulations of non-equilibrium thermal transport were conducted in a three-dimensional device-like silicon structure, discretized using 604,054 tetrahedral cells. The angular space was discretized using 400 angles, and the spectral space was discretized into 40 spectral intervals (bands). This resulted in ∼9.7 × 109 unknowns, which are approximately 3 orders of magnitude larger than previously reported computations in this area. Studies showed that direction-based and hybrid band/cell-based parallelization strategies resulted in similar total computational time. However, the parallel efficiency of the hybrid band/cell-based strategy—about 88%—was found to be superior to that of the direction-based strategy, and is recommended as the preferred strategy for even larger scale computations.

Impact of mesh partitioning methods in CFD for large scale parallel computing

It was well known that high performance computing depends on load balance and mesh cell distributions. A well balanced load can improve efficiency of parallel computing and minimizing the number of cells with remote neighbors can reduce the time spent on the data communication among processors. Mesh partitioning methods were able to optimize the mesh cell distributions for large scale parallel computing. In this paper, the open source mesh partitioning software packages, such as Metis, ParMetis, Scotch, PT-Scotch and Zoltan, were ported into Code_Saturne, which is an open source CFD software package, for realizing the optimization on large scale CFD applications. Through the studies, it was found that the mesh partitioning methods strongly affect the CFD code on high performance computing especially for large scale parallel computing.

Power-law spectra formed by stretching polymers in decaying isotropic turbulence

The spectral dynamics of isotropic decaying turbulence with polymer additives is numerically investigated using a hybrid Eulerian-Lagrangian approach with making use of large-scale parallel computation. We found that the kinetic energy and pressure variance spectra obeyed the power law E(k) ∼ k−α and Ep(k) ∼ k−β in the scale range below the Kolmogorov length lK when the turbulence sufficiently decayed while the Weissenberg number Wi remained greater than unity. The exponents α and β were found to be α = 4.1 − 4.6 and β = 2.8 − 3.2, respectively, and were found to decrease with increasing Wi. We discuss the similarities and differences between the present results and the results of previous experimental and numerical studies for elastic turbulence, which is characterized by Wi ≫ 1 and a Reynolds number below unity.

Detecting silent data corruption through data dynamic monitoring for scientific applications

Parallel programming has become one of the best ways to express scientific models that simulate a wide range of natural phenomena. These complex parallel codes are deployed and executed on large-scale parallel computers, making them important tools for scientific discovery. As supercomputers get faster and larger, the increasing number of components is leading to higher failure rates. In particular, the miniaturization of electronic components is expected to lead to a dramatic rise in soft errors and data corruption. Moreover, soft errors can corrupt data silently and generate large inaccuracies or wrong results at the end of the computation. In this paper we propose a novel technique to detect silent data corruption based on data monitoring. Using this technique, an application can learn the normal dynamics of its datasets, allowing it to quickly spot anomalies. We evaluate our technique with synthetic benchmarks and we show that our technique can detect up to 50% of injected errors while incurring only negligible overhead.

Parallel molecular computation of modular-multiplication with two same inputs over finite field GF(2n) using self-assembly of DNA tiles

Two major advantages of DNA computing – huge memory capacity and high parallelism – are being explored for large-scale parallel computing, mass data storage and cryptography. Tile assembly model is a highly distributed parallel model of DNA computing. Finite field GF(2n) is one of the most commonly used mathematic sets for constructing public-key cryptosystem. It is still an open question that how to implement the basic operations over finite field GF(2n) using DNA tiles. This paper proposes how the parallel tile assembly process could be used for computing the modular-square, modular-multiplication with two same inputs, over finite field GF(2n). This system could obtain the final result within less steps than another molecular computing system designed in our previous study, because square and reduction are executed simultaneously and the previous system computes reduction after calculating square. Rigorous theoretical proofs are described and specific computing instance is given after defining the basic tiles and the assembly rules. Time complexity of this system is 3n−1 and space complexity is 2n2.

Analysis of Rolling Contact Behavior between Wheel and Rail through Large-scale Parallel Computing

This paper attempts to analyze dynamic wheel/rail rolling contact through application of a three-dimensional finite element method with large-scale parallel computing. This analysis clarifies the dynamic behavior of the contact patch, when a wheel rolls on a rail at a high speed with loading torque. A rail model which allows efficient calculation was developed to accelerate a wheel on a small size rail model. In addition, this method employs dynamic mesh partitioning to maintain meshes in the contact patch in one partition region during the wheel rolling for efficient parallel calculation. Using this method has made it possible to evaluate the dynamic wheel/rail rolling contact behavior.

非線形長波モデルと流体粒子法による津波シミュレータの開発

We develop a tsunami simulator integrating a 3-D fluid simulation technology that runs on large-scale parallel computers using smoothed-particle hydrodynamics (SPH) method, together with a 2-D tsunami propagation simulation technique using a nonlinear shallow water wave model. We use the 2-D simulation to calculate tsunami propagation of scale of about 1000km from epicenter to near shore. The 3-D SPH method can be used to calculate the force that a tsunami can exert on a building, and to simulate flooding patterns in urban area of at most km scale. By applying the processing power of computers to the technologies resulting from this research, we seek to contribute to improved disaster preparedness and disaster mitigation through a better understanding of tsunami's mechanisms.

Implementation and Evaluation of an AMR Framework for FDM Applications

In order to e xecute various finite-difference method applications on large-scale parallel computers with a reasonable cost of computer resources, a framework using an adaptive mesh refinement (AMR) technique has been developed. AMR can realize high -resolution simulations while saving computer resources by generating and removing hierarchica l grids dynamically. In the AMR frame work, a dynamic domain decomposition (DDD) technique, as a dynamic load balancing method, is also implemented to correct the computational load imbalance between each process associated with parallelization. By performing a 3D AM R test simulation, it is confirmed that dynamic load balancing can be achieved and execution time can be reduced by introducing the DDD technique.

A213 OHP解析に向けた大規模並列向け気液二相流解析法の開発

A213 OHP解析に向けた大規模並列向け気液二相流解析法の開発