Hybrid Parallel Method Research Articles

Distributed Implementation of Computing Language Model in Cloud Computing Network

In this paper, the distributed implementation of Computing Language Model (CLM) in cloud computing network is studied and discussed. With the advent of the era of big data, the application of CLM in natural language processing (NLP), machine translation and other fields is increasingly extensive, and the demand for computing resources is also increasing. As an effective way to manage computing resources, distributed computing can make full use of resources in cloud computing environment and realize efficient execution of computing tasks. In this paper, the application of distributed computing methods such as data parallelism, model parallelism and hybrid parallelism in cloud computing environment is studied and analyzed, and the advantages and disadvantages of different methods in CLM implementation are discussed. The experimental results show that the hybrid parallel method can effectively combine the advantages of data parallelism and model parallelism, and improve the training efficiency and performance of CLM. This study provides important theoretical guidance and technical support for the distributed implementation of CLM in cloud computing networks, and is of great significance for further promoting the development of big data and AI technology.

Alternating directions parallel hybrid memory iGRM direct solver for non-stationary simulations

The three-dimensional isogeometric analysis (IGA-FEM) is a modern method for simulation. The idea is to utilize B-splines or NURBS basis functions for both computational domain descriptions and the engineering computations. Refined isogeometric analysis (rIGA) employs a mixture of patches of elements with B-spline basis functions, and $C^0$ separators between them. It enables a reduction of the computational cost of direct solvers. Both IGA and rIGA come with challenging sparse matrix structure, that is expensive to generate. In this paper, we show a hybrid parallelization method to reduce the computational cost of the integration phase using hybrid-memory parallel machines. The two-level parallelization includes the partitioning of the computational mesh into sub-domains on the first level (MPI), and loop parallelization on the second level (OpenMP). We show that hybrid parallelization of the integration reduces the contribution of this phase significantly. Thus, alternative algorithms for fast isogeometric integration are not necessary.

A hybrid MPI-CUDA approach for nonequispaced discrete Fourier transformation

Nonequispaced discrete Fourier transformation (NDFT) is widely applied in all aspects of computational science and engineering. The computational efficiency and accuracy of NDFT has always been a critical issue in hindering its comprehensive applications both in intensive and in extensive aspects of scientific computing. In our previous work Yang et al. (2018), a CUNFFT method was proposed and it shown outstanding performance in handling NDFT at intermediate scale based on CUDA (Compute Unified Device Architecture) technology. In the current work, we further improved the computational efficiency of the CUNTTF method using an efficient MPI-CUDA hybrid parallelization (HP) scheme of NFFT to achieve a cutting-edge treatment of NDFT at super extended scale. Within this HP-NFFT method, the spatial domain of NDFT is decomposed into several parts according to the accumulative feature of NDFT and the detailed number of CPU and GPU nodes. These decomposed NDFT subcells are independently calculated on different CPU nodes using a MPI process-level parallelization mode, and on different GPU nodes using a CUDA thread-level parallelization mode and CUNFFT algorithm. A massive benchmarking of the HP-NFFT method indicates that this method exhibits a dramatic improvement in computational efficiency for handling NDFT at super extended scale without loss of computational precision. Furthermore, the HP-NFFT method is validated via the calculation of Madelung constant of fluorite crystal structure, and thereafter verified that this method is robust for the calculation of electrostatic interactions between charged ions in molecular dynamics simulation systems. Program summaryProgram title: HP-NFFTCPC Library link to program files:http://dx.doi.org/10.17632/ys2y92jkwy.1Licensing provisions: GNU General Public License 3Programming language: MPI, C, and CUDA CSupplementary material: The program is designed for effective computation of large-scale nonequispaced discrete Fourier transformation (NDFT), which runs on particular computers equipped with NVIDIA GPUs. It has been tested on (a) one single computer node with Intel(R) Core(TM) i7-3770 @ 3.40 GHz (CPU) and GTX 980 Ti (GPU), and (b) MPI parallel computer nodes with the same configurations.Nature of problem: For NDFT, the computation is extremely time-consuming in many domains of computational physics due to the failure to utilize FFT directly, which often affects the computational efficiency of whole system seriously. Although the parallel method CUNFFT based on GPU has achieved a qualitative leap compared with previous methods in NDFT computation, the computation capability is limited to the throughput capacity of GPU when the size of NDFT system is big enough.Solution method: We constructed a hybrid parallel architecture, in which CPU and GPU are combined to accelerate the NDFT computation effectively. Firstly, the NDFT system is divided into several subcells via domain-decomposition method. Then MPI (Message Passing Interface) is used to implement the CPU-parallel computation with each computer node corresponding to a particular subcell, and furthermore each subcell in one computer node will be executed in GPU in parallel efficiently. In this hybrid parallel method, the most critical technical problem is how to parallelize a CUNFFT in the parallel strategy, which is conquered effectively by deep-seated research of basic principles and some algorithm skills.Restrictions: The HP-NFFT is mainly oriented to, and has shown significant computational efficiency to deal with large-scale NDFT computations. However, for a small NDFT system containing less than 10^6 particles, the mode of multiple computer nodes has no apparent efficiency advantage or even lower efficiency compared with the mode of a single computer node because of the serious network delay.

Implementation of the CPU/GPU hybrid parallel method of characteristics neutron transport calculation using the heterogeneous cluster with dynamic workload assignment

In recent years, graphics processing units (GPUs) have been adopted in many High-Performance Computing (HPC) systems due to their massive computational power and superior energy efficiency. And accelerating CPU-version computational code on heterogeneous clusters with multi-core CPUs and GPUs has attracted a lot of attention. One of the focus on heterogeneous computing is to efficiently take advantage of all computational resources, including both CPU and GPU available on a cluster. In this paper, a heterogeneous MPI + OpenMP/CUDA parallel algorithm for solving the 2D neutron transport equation with the method of characteristic (MOC) is implemented. In this algorithm, the spatial domain decomposition technique provides the coarse-grained parallelism with the MPI protocol while the fine-grained parallelism is exploited through OpenMP (in CPU calculated domain) and CUDA (in GPU calculated domain) based on the ray parallelization. In order to efficiently leverage the computing power of heterogeneous clusters, a dynamic workload assignment scheme is proposed, which is to distribute the workload based on the runtime performance of CPUs and GPUs in the cluster. Moreover, the strong scaling performance of the MPI + CUDA parallelization is studied through a performance analysis model which provides the detailed impact of the degradation in iteration scheme, the load imbalance issue, the data copy between CPUs and GPUs, and the MPI communication in the MPI + CUDA parallel algorithm. And the corresponding conclusion is still tenable for the MPI + OpenMP/CUDA parallelization. The C5G7 2D benchmark and an extended 2D whole-core problem are calculated with MPI + CUDA parallelization, MPI + OpenMP/CUDA parallelization, and the MPI parallelization for comparison. Numerical results demonstrate that the heterogeneous parallel algorithm maintains the desired accuracy. And the dynamic workload assignment scheme can provide the optimal workload assignment which ideally matches the experimental results. In addition, over 11% improvement is observed in MPI + OpenMP/CUDA parallelization compared against the MPI + CUDA parallelization. Moreover, the CPUs/GPUs heterogeneous clusters significantly outperform the CPUs clusters and one heterogeneous node shows basically five times faster than a CPUs node.

Speeding up multi-objective optimization of liquid fossil fuel reserve exploitation with parallel hybrid memory integration

In this paper, we show a hybrid parallelization method for reducing the computational cost of a solver using hybrid-memory parallel machines. We show how the hybrid parallelization can be utilized to speed up challenging computational applications. Namely, we focus on the liquid fossil fuel reservoir exploitation problem (LFFEP), the optimization inverse problem where we try to find the optimal locations of pumps and sinks during the oil exploitation technique with hydraulic fracturing to maximize the amount of oil and minimize groundwater contamination. In our simulations, we combine a hierarchical genetic search (HGS) with an isogeometric finite element method solver.

A Hybrid Parallel Method for 3-D Nonlinear Periodic Eddy Current Problems With Motions

This paper describes a highly robust and efficient hybrid parallel computing method for periodic eddy current problems considering motions. This hybrid parallel approach is composed of two levels of parallelization, with distributed memory parallel based on message passing interface at the higher level and shared memory parallel based on multi-threading at the lower level. Application examples will be presented to demonstrate the effectiveness of this method.

A hybrid parallel architecture for electrostatic interactions in the simulation of dissipative particle dynamics

In this work, we upgraded the electrostatic interaction method of CU-ENUF (Yang, et al., 2016) which first applied CUNFFT (nonequispaced Fourier transforms based on CUDA) to the reciprocal-space electrostatic computation and made the computation of electrostatic interaction done thoroughly in GPU. The upgraded edition of CU-ENUF runs concurrently in a hybrid parallel way that enables the computation parallelizing on multiple computer nodes firstly, then further on the installed GPU in each computer. By this parallel strategy, the size of simulation system will be never restricted to the throughput of a single CPU or GPU. The most critical technical problem is how to parallelize a CUNFFT in the parallel strategy, which is conquered effectively by deep-seated research of basic principles and some algorithm skills. Furthermore, the upgraded method is capable of computing electrostatic interactions for both the atomistic molecular dynamics (MD) and the dissipative particle dynamics (DPD). Finally, the benchmarks conducted for validation and performance indicate that the upgraded method is able to not only present a good precision when setting suitable parameters, but also give an efficient way to compute electrostatic interactions for huge simulation systems. Program summaryProgram title: HP-ENUFProgram Files doi:http://dx.doi.org/10.17632/zncf24fhpv.1Licensing provisions: GNU General Public License 3 (GPL)Programming language: C, C++, and CUDA CSupplementary material:The program is designed for effective electrostatic interactions of large-scale simulation systems, which runs on particular computers equipped with NVIDIA GPUs. It has been tested on (a) single computer node with Intel(R) Core(TM) i7-3770@ 3.40 GHz (CPU) and GTX 980 Ti (GPU), and (b) MPI parallel computer nodes with the same configurations.Nature of problem:For molecular dynamics simulation, the electrostatic interaction is the most time-consuming computation because of its long-range feature and slow convergence in simulation space, which approximately take up most of the total simulation time. Although the parallel method CU-ENUF (Yang et al., 2016) based on GPU has achieved a qualitative leap compared with previous methods in electrostatic interactions computation, the computation capability is limited to the throughput capacity of a single GPU for super-scale simulation system. Therefore, we should look for an effective method to handle the calculation of electrostatic interactions efficiently for a simulation system with super-scale size.Solution method:We constructed a hybrid parallel architecture, in which CPU and GPU are combined to accelerate the electrostatic computation effectively. Firstly, the simulation system is divided into many subtasks via domain-decomposition method. Then MPI (Message Passing Interface) is used to implement the CPU-parallel computation with each computer node corresponding to a particular subtask, and furthermore each subtask in one computer node will be executed in GPU in parallel efficiently. In this hybrid parallel method, the most critical technical problem is how to parallelize a CUNFFT (nonequispaced fast Fourier transform based on CUDA) in the parallel strategy, which is conquered effectively by deep-seated research of basic principles and some algorithm skills.Restrictions:The HP-ENUF is mainly oriented to super-scale system simulations, in which the performance superiority is shown adequately. However, for a small simulation system containing less than 106 particles, the mode of multiple computer nodes has no apparent efficiency advantage or even lower efficiency due to the serious network delay among computer nodes, than the mode of single computer node.

Hybrid parallel approach to homogenization of transport processes in masonry

Numerical simulations of transport processes, e.g., heat and moisture transport, in heterogeneous masonry structures performed on single processor computers are very time consuming. The classical description by the finite element method leads to enormous numbers of degrees of freedom. It is caused by the fact that there are relatively thin layers of mortar in contrary to large bricks or stones. The mortar and stones have very different material parameters and the finite element mesh has to be able to describe the temperature and moisture fields in the thin layers of mortar and in their vicinity. An application of a multi-scale approach in connection with parallel computing can be successfully exploited. The whole structure is described by a reasonably coarse finite element mesh, called the macro-scale problem, and the material parameters are obtained from the lower-level problems, called the meso-scale problem, by a homogenization procedure. In this procedure, the macro-problem is assigned to the master processor and the meso-scale problems belong to the slave processors. This proposed approach is called hybrid parallel method.

Hybrid Parallel Method for XML Parsing

This paper presents a hybrid parallel method for XML parsing,which consists of a lightweight events partition stage,followed by an event-level parallel parsing stage,and a final post-processing stage.SIMD instructions are used to speed up the processing in the events partition stage.Software pipelined processing is achieved at stage level.The study combined event-level data parallel parsing technique and pipelined processing technique to create a hybrid parallel method.Compared to other parallel solutions,the method has the advantage of a much more efficient parallel processing with low synchronization overhead.The method is tested on a Linux machine with Intel Xeon X7560 CPU for 8 cores,and the results show the method can achieve a much higher speed up and better scalability than other software implementations done to date.

A hybrid, massively parallel implementation of a genetic algorithm for optimization of the impact performance of a metal/polymer composite plate

A hybrid parallelization method composed of a coarse-grained genetic algorithm (GA) and fine-grained objective function evaluations is implemented on a heterogeneous computational resource consisting of 16 IBM Blue Gene/P racks, a single x86 cluster node and a high-performance file system. The GA iterator is coupled with a finite-element (FE) analysis code developed in house to facilitate computational steering in order to calculate the optimal impact velocities of a projectile colliding with a polyurea/structural steel composite plate. The FE code is capable of capturing adiabatic shear bands and strain localization, which are typically observed in high-velocity impact applications, and it includes several constitutive models of plasticity, viscoelasticity and viscoplasticity for metals and soft materials, which allow simulation of ductile fracture by void growth. A strong scaling study of the FE code was conducted to determine the optimum number of processes run in parallel. The relative efficiency of the hybrid, multi-level parallelization method is studied in order to determine the parameters for the parallelization. Optimal impact velocities of the projectile calculated using the proposed approach, are reported.

하이브리드 병렬 프로그램을 이용한 타키온 슈퍼컴퓨터의 성능

Tachyon primary system which introduces recently is a high performance supercomputer that composed with AMD Barcelona nodes. In this paper, we will verify the performance and parallel scalability of TachyonIn by using multi-zone NAS Parallel Benchmark(NPB) which is one of a program with hybrid parallel method. To test performance of hybrid parallel execution, B and C classes of BT-MZ in NPB version 3.3 were used. And the parallel scalability test has finished with Tachyon`s 1024 processes. It is the first time in Korea to get a result of hybrid parallel computing calculation using more than 1024 processes. Hybrid parallel method in high performance computing system with multi-core technology like Tachyon describes that it can be very efficient and useful parallel performance benchmarks.

Performance evaluation of load balancing strategies for approximate string matching application on an MPI cluster of heterogeneous workstations

In this paper, we present three parallel approximate string matching methods on a parallel architecture with heterogeneous workstations to gain supercomputer power at low cost. The first method is the static master–worker with uniform distribution strategy, the second one is the dynamic master–worker with allocation of subtexts and the third one is the dynamic master–worker with allocation of text pointers. Further, we propose a hybrid parallel method that combines the advantages of static and dynamic parallel methods in order to reduce the load imbalance and communication overhead. This hybrid method is based on the following optimal distribution strategy: the text collection is distributed proportional to workstation’s speed. We evaluated and compared the performance of the four methods with clusters one, two, four, six and eight heterogeneous workstations. The experimental results demonstrate that dynamic allocation of text pointers and hybrid methods achieve better performance than the two original ones. We also present an analytical performance model for the four methods that confirms the actual behaviour of the experimental results.