Performance Of Parallel Algorithm Research Articles

Impact of multi-GPUs and hyperparameters on parallel algorithms based on pytorch distributed

Neural networks, as powerful models for solving many genuine problems, are suffering from the issue of being too computationally intensive due to the excessive size of the models themselves and the large datasets used for training. This paper’s work focuses on a parallel algorithm, experimented on the classical convolutional neural network LeNet-5, aiming to reduce the time it takes for the neural network to train. This applies a massively parallel GPU structure to take full advantage of its high computing power. Among the approaches to improve the performance of parallel algorithms, many efforts have been made to improve the design of parallel structures, while this work focuses more on the enhancement of parallel algorithms by reasonable settings of neural network hyperparameters. This paper tries to approach (1) reducing the running time by testing parallelism with different numbers of GPUs and (2) further investigating the relationship between parallelism and model accuracy degradation and experimentally deriving a parallel scheme that can reduce the training time while guaranteeing accuracy.

Parallel multi-speed Pursuit-Evasion Game algorithms

Pursuit-Evasion Game (PEG) consists of a team of pursuers trying to capture one or more evaders. PEG is important due to its application in surveillance, search and rescue, disaster robotics, boundary defense and so on. In general, PEG requires exponential time to compute the minimum number of pursuers to capture an evader. To mitigate this, we have designed a parallel optimal algorithm to minimize the capture time in PEG. Given a discrete topology, this algorithm also outputs the minimum number of pursuers to capture an evader. We also extended parallel algorithm to consider other versions as: heterogeneous/multi-speed players; the pac-dot technique to increase evader lifetime in a game; and a pruning strategy for pac-dot technique to increase the scalability. The parallel algorithm performance was evaluated by speedup metric and this algorithm and its extensions were simulated and evaluated on many different topologies to validate the viability of our algorithms by discussing and evaluating a set of simulation results. The parallel algorithm enables us to scale up to 8.13 times with 8 cores compared to state-of-the-art. Considering the complexity of the state space growing up, pruning technique to pac-dot algorithm minimizes the state space and generated transition, and can handle a large number of states (≈830 M) and transitions (≈11 G) generated. In general, our algorithms increase the scalability and make it feasible to compute the PEG optimal strategy for more realistic cases.

English

Evaluating how well a whole system or set of subsystems performs is one of the primary objectives of performance testing. We can tell via performance assessment if the architecture implementation meets the design objectives. Performance evaluations of several parallel algorithms are compared in this study. Both theoretical and experimental methods are used in performance assessment as a subdiscipline in computer science. The parallel method outperforms its sequential counterpart in terms of throughput. The parallel algorithm's performance (speedup) is examined, as shown in the result.

A new parallel algorithm for improving the computational efficiency of multi-GNSS precise orbit determination

The computational efficiency is critical with the increasing number of GNSS satellites and ground stations since many unknown parameters must be estimated. Although only active parameters are kept in the normal equation in sequential least square estimation, the computational cost for parameter elimination is still a heavy burden. Therefore, it is necessary to optimize the procedure of parameter elimination to enhance the computational efficiency of GNSS network solutions. An efficient parallel algorithm is developed for accelerating parameter estimation based on modern multi-core processors. In the parallel algorithm, a multi-thread guided scheduling scheme, and cache memory traffic optimizations are implemented in parallelized sub-blocks for normal-equation-level operations. Compared with the traditional serial scheme, the computational time of parameter estimations can be reduced by a factor of three due to the new parallel algorithm using a six-core processor. Our results also confirm that the architecture of computers entirely limits the performance of the parallel algorithm. All the parallel optimizations are also investigated in detail according to the characteristics of CPU architecture. This gives a good reference to architecture-oriented parallel programming in the future development of GNSS software. The performance of the multi-thread parallel algorithm is expected to improve further with the upgrade of new multi-core coprocessors.s

High performance ant colony system based on GPU warp specialization with a static–dynamic balanced candidate set strategy

To improve the performance of parallel algorithms, it is necessary to make full utilization of computing resources and computing power of parallel hardware. However, the utilization efficiency of the computation must also be considered. Ant Colony System (ACS) has natural parallelism, and the procedure “Selecting the next element” is one of its main key computational components for combinatorial optimization problems. If all elements in the candidate set have been visited, a global search in the complete element domain needs to be performed and its computational overhead is enormous. Based on extensive experiments, we show that the results of global searches are also valuable for subsequent iterations of ACS. Therefore, an innovative static–dynamic balanced candidate set strategy, denoted by ID-CS, is proposed. ID-CS saves the result of global searches in previous iterations in order to reuse them in later iterations so that it can decrease the number of global searches. Furthermore, a novel GPU parallel ACS algorithm, ACS_GPU_WSP, is proposed based on the producer–consumer parallel model by GPU Warp Specialization. Each ant divides its computational work into private work and public work. For 11 large-scale typical TSP problems, compared with the state-of-art GPU data-level parallel implementation, it has achieved a large performance improvement.

Operators for Data Redistribution: Applications to the STL Library and RayTracing Algorithm

In distributed-memory systems, data redistributions are operations that change the ownership and location of a selected subset of a data structure at runtime. They allow the improvement of the performance of parallel algorithms which operate on changing or partial domains, aiming to create a balanced workload among the active processes. To manually redistribute data is a cumbersome and error-prone task. In this paper, we present a method based on four combinable operators to redistribute partial domains selected by the programmer at runtime in an efficient and simple way. They abstract to the programmer the data-redistribution implementation details, such as the new mapping, relocation, and communication of the selected data. We also present the application of the proposed operators to a RayTracing application and to a significant part of STL (C++ Standard Template Library). Our experimental results show that our approach automatically generates a good load balance, which leads to performance improvements for generic data-distribution policies. It does not introduce significant performance overheads compared with tailored data redistributions directly programmed using MPI (Message Passing Interface), while it greatly reduces the code development effort.

Параллельное решение систем линейных уравнений на гибридной архитектуре CPU+GPU

The article discusses the parallel implementation of solving systems of linear algebraic equations on computational nodes containing a central processing unit (CPU) and graphic accelerators (GPU). The performance of parallel algorithms for the classical conjugate gradient method schemes when using the CPU and GPU together is significantly limited by the synchronization points. The article investigates the pipeline version of the conjugate gradient method with one synchronization point, the possibility of asynchronous calculations, load balancing between the CPU and GPU when solving the large linear systems. Numerical experiments were carried out on test matrices and computational nodes of different performance of a heterogeneous cluster, which allowed us to estimate the contribution of communication costs. The algorithms are implemented with the joint use of technologies: MPI, OpenMP and CUDA. The proposed algorithms, in addition to reducing the execution time, allow solving large linear systems, for which there are not enough memory resources of one GPU or a computing node. At the same time, block algorithm with the pipelining decreases the total execution time by reducing synchronization points and aggregating some messages in one.

On the performance difference between theory and practice for parallel algorithms

The performance of parallel algorithms is often inconsistent with their preliminary theoretical analyses. Indeed, the difference is increasing between the ability to theoretically predict the performance of a parallel algorithm and the results measured in practice. This is mainly due to the accelerated development of advanced parallel architectures, whereas there is still no agreed model for parallel computation, which has implications for the design of parallel algorithms and for the manner in which parallel programming should be taught.In this study, we examined the practical performance of Cormen’s Quicksort parallel algorithm. We determined the performance of the algorithm with different parallel programming approaches and examine the capacity of theoretical performance analyses of the algorithm for predicting the actual performance. This algorithm is used for teaching theoretical and practical aspects of parallel programming to undergraduate students. We considered the pedagogic implications that may arise when the algorithm is used as a learning resource for teaching parallel programming.

A Three-Dimensional Cartesian Mesh Generation Algorithm Based on the GPU Parallel Ray Casting Method

Robust and efficient Cartesian mesh generation for large-scale scene is of great significance for fluid dynamics simulation and collision detection. High-quality and large-scale mesh generation task in a personal computer is hard to achieve. In this paper, a parallel Cartesian mesh generation algorithm based on graphics processing unit (GPU) is proposed. The proposed algorithm is optimized based on the traditional ray casting method in computer graphics, and is more efficient and stable for large-scale Cartesian mesh generation. In the process of mesh generation, the geometries represented by triangular facets are transformed into a mesh composed of orthogonal hexahedrons. A parallel ray generation method is proposed to reduce the data exchange between the host memory and device memory. A parallel primitives searching method based on lattice grid is adopted to search the triangular facets for intersection calculation between rays and triangles. The parallel Cartesian mesh generation algorithm has been implemented using CUDA library. The performance of parallel Cartesian mesh generation algorithm has been promoted enormously compared with the traditional the sequential algorithm, which is shown in different numerical experiments. Through some tests, the performance of parallel algorithm is analyzed, and the results show that the parallel computing power of the GPU is fully utilized. Finally, examples of Cartesian mesh generation are presented.

Numerical analysis of parallel implementation of the reorthogonalized ABS methods

Solving systems of equations is a critical step in various computational tasks. We recently published a novel ABS-based reorthogonalization algorithm to compute the QR factorization. Experimental analysis on the Matlab 2015a platform revealed that this new ABS-based algorithm was able to more accurately calculate the rank of the coefficient matrix, the determination of the orthogonal bases and the QR factorization than the built-in rank or qr Matlab functions. However, the reorthogonalization process significantly increased the computation cost. Therefore, we tested a new approach to accelerate this algorithm by implementing it on different parallel platforms. The above mentioned ABS-based reorthogonalization algorithm was implemented using Matlab’s parallel computing toolbox and accelerated massive parallelism (AMP) runtime library. We have tested various matrices including Pascal, Vandermonde and randomly generated dense matrices. The performance of the parallel algorithm was determined by calculating the speed-up factor defined as the fold reduction of execution time compared to the sequential algorithm. For comparison, we also tested the effect of parallel implementation of the classical Gram–Schmidt algorithm incorporating a reorthogonalization step. The results show that the achieved speed-up is significant, and also the performance of this practical parallel algorithm increases as the number of equations grows. The results reveal that the reorthogonalized ABS algorithm is practical and efficient. This fact expands the practical usefulness of our algorithms.

Let’s HPC: A web-based platform to aid parallel, distributed and high performance computing education

Let’s HPC (www.letshpc.org) is an evolving open-access web-based platform to supplement conventional classroom oriented High Performance Computing (HPC) and Parallel & Distributed Computing (PDC) education. This platform has been developed to allow users to learn, evaluate, teach and see the performance of parallel algorithms from a system’s viewpoint. The Let’s HPC platform’s motivation comes from the experiences of teaching HPC/PDC courses and it is designed to help streamline the process of analyzing parallel programs.At the heart of this platform is a database archiving the performance and execution environment related data of standard parallel algorithm implementations run on different computing architectures using different programming environments. The online plotting and analysis tools of our platform can be combined seamlessly with the database to aid self-learning, teaching, evaluation and discussion of different HPC related topics, with a particular focus on a holistic system’s perspective. The user can quantitatively compare and understand the importance of numerous deterministic as well as non-deterministic factors of both the software and the hardware that impact the performance of parallel programs. Instructors of HPC/PDC related courses can use the platform’s tools to illustrate the importance of proper data collection and analysis in understanding factors impacting performance as well as to encourage peer learning among students. Scripts are provided for automatically collecting performance related data, which can then be analyzed using the platform’s tools. The platform also allows students to prepare a standard lab/project report aiding the instructor in uniform evaluation. The platform’s modular design enables easy inclusion of performance related data from contributors as well as addition of new features in the future.This paper summarizes the background and motivation behind the Let’s HPC project, the design philosophy of the platform, the present capabilities of the platform, as well as the plans for future developments.

Multi-GPU parallel algorithm design and analysis for improved inversion of probability tomography with gravity gradiometry data

In this paper, we make a study on the inversion of probability tomography (IPT) with gravity gradiometry data at first. The space resolution of the results is improved by multi-tensor joint inversion, depth weighting matrix and the other methods. Aiming at solving the problems brought by the big data in the exploration, we present the parallel algorithm and the performance analysis combining Compute Unified Device Architecture (CUDA) with Open Multi-Processing (OpenMP) based on Graphics Processing Unit (GPU) accelerating. In the test of the synthetic model and real data from Vinton Dome, we get the improved results. It is also proved that the improved inversion algorithm is effective and feasible. The performance of parallel algorithm we designed is better than the other ones with CUDA. The maximum speedup could be more than 200. In the performance analysis, multi-GPU speedup and multi-GPU efficiency are applied to analyze the scalability of the multi-GPU programs. The designed parallel algorithm is demonstrated to be able to process larger scale of data and the new analysis method is practical.

Tuning the granularity of parallelism for distributed graph processing

Popular distributed graph processing frameworks, such as Pregel and GraphLab, are based on the vertex-centric computation model, where users write their customized Compute function for each vertex to process the data iteratively. Vertices are evenly partitioned among the compute nodes, and the granularity of parallelism of the graph algorithm is normally tuned by adjusting the number of compute nodes. Vertex-centric model splits the computation into phases. Inside one specific phase, the computation proceeds as an embarrassingly parallel process, because no communication between compute nodes incurs. By default, current graph engine only handles one iteration of the algorithm in a phase. However, in this paper, we find that we can also tune the granularity of parallelism, by aggregating the computation of multiple iterations into one phase, which has a significant impact on the performance of the graph algorithm. In the ideal case, if all computations are handled in one phase, the whole algorithm turns into an embarrassingly parallel algorithm and the benefit of parallelism is maximized. Based on this observation, we propose two approaches, a function-based approach and a parameter-based approach, to automatically transform a Pregel algorithm into a new one with tunable granularity of parallelism. We study the cost of such transformation and the trade-off between the granularity of parallelism and the performance. We provide a new direction to tune the performance of parallel algorithms. Finally, the approaches are implemented in our graph processing system, N2, and we illustrate their performance using popular graph algorithms.

MPI/OpenMP hybrid parallel algorithm for resolution of identity second-order Møller-Plesset perturbation calculation of analytical energy gradient for massively parallel multicore supercomputers.

A massively parallel algorithm of the analytical energy gradient calculations based the resolution of identity Møller-Plesset perturbation (RI-MP2) method from the restricted Hartree-Fock reference is presented for geometry optimization calculations and one-electron property calculations of large molecules. This algorithm is designed for massively parallel computation on multicore supercomputers applying the Message Passing Interface (MPI) and Open Multi-Processing (OpenMP) hybrid parallel programming model. In this algorithm, the two-dimensional hierarchical MP2 parallelization scheme is applied using a huge number of MPI processes (more than 1000 MPI processes) for acceleration of the computationally demanding O(N5 ) step such as calculations of occupied-occupied and virtual-virtual blocks of MP2 one-particle density matrix and MP2 two-particle density matrices. The new parallel algorithm performance is assessed using test calculations of several large molecules such as buckycatcher C60 @C60 H28 (144 atoms, 1820 atomic orbitals (AOs) for def2-SVP basis set, and 3888 AOs for def2-TZVP), nanographene dimer (C96 H24 )2 (240 atoms, 2928 AOs for def2-SVP, and 6432 AOs for cc-pVTZ), and trp-cage protein 1L2Y (304 atoms and 2906 AOs for def2-SVP) using up to 32,768 nodes and 262,144 central processing unit (CPU) cores of the K computer. The results of geometry optimization calculations of trp-cage protein 1L2Y at the RI-MP2/def2-SVP level using the 3072 nodes and 24,576 cores of the K computer are presented and discussed to assess the efficiency of the proposed algorithm. © 2017 Wiley Periodicals, Inc.

All Phase Discrete Sine Biorthogonal Transform and Its Application in JPEG-like Image Coding Using GPU

Discrete cosine transform (DCT) based JPEG standard significantly improves the coding efficiency of image compression, but it is unacceptable event in serious blocking artifacts at low bit rate and low efficiency of high-definition image. In the light of all phase digital filtering theory, this paper proposes a novel transform based on discrete sine transform (DST), which is called all phase discrete sine biorthogonal transform (APDSBT). Applying APDSBT to JPEG scheme, the blocking artifacts are reduced significantly. The reconstructed image of APDSBT-JPEG is better than that of DCT-JPEG in terms of objective quality and subjective effect. For improving the efficiency of JPEG coding, the structure of JPEG is analyzed. We analyze key factors in design and evaluation of JPEG compression on the massive parallel graphics processing units (GPUs) using the compute unified device architecture (CUDA) programming model. Experimental results show that the maximum speedup ratio of parallel algorithm of APDSBT-JPEG can reach more than 100 times with a very low version GPU. Some new parallel strategies are illustrated in this paper for improving the performance of parallel algorithm. With the optimal strategy, the efficiency can be improved over 10%.

Parallel computations in nonlinear solid mechanics using adaptive finite element and meshless methods

Purpose – A variety of meshless methods have been developed in the last 20 years with an intention to solve practical engineering problems, but are limited to small academic problems due to associated high computational cost as compared to the standard finite element methods (FEM). The purpose of this paper is to develop an efficient and accurate algorithms based on meshless methods for the solution of problems involving both material and geometrical nonlinearities. Design/methodology/approach – A parallel two-dimensional linear elastic computer code is presented for a maximum entropy basis functions based meshless method. The two-dimensional algorithm is subsequently extended to three-dimensional adaptive nonlinear and three-dimensional parallel nonlinear adaptively coupled finite element, meshless method cases. The Prandtl-Reuss constitutive model is used to model elasto-plasticity and total Lagrangian formulations are used to model finite deformation. Furthermore, Zienkiewicz and Zhu and Chung and Belytschko error estimation procedure are used in the FE and meshless regions of the problem domain, respectively. The message passing interface library and open-source software packages, METIS and MUltifrontal Massively Parallel Solver are used for the high performance computation. Findings – Numerical examples are given to demonstrate the correct implementation and performance of the parallel algorithms. The agreement between the numerical and analytical results in the case of linear elastic example is excellent. For the nonlinear problems load-displacement curve are compared with the reference FEM and found in a very good agreement. As compared to the FEM, no volumetric locking was observed in the case of meshless method. Furthermore, it is shown that increasing the number of processors up to a given number improve the performance of parallel algorithms in term of simulation time, speedup and efficiency. Originality/value – Problems involving both material and geometrical nonlinearities are of practical importance in many engineering applications, e.g. geomechanics, metal forming and biomechanics. A family of parallel algorithms has been developed in this paper for these problems using adaptively coupled finite element, meshless method (based on maximum entropy basis functions) for distributed memory computer architectures.

Parallelized methods for solving polynomial equations

The current microprocessors are concentrating on the multiprocessor or multi - core system architecture. The parallel algorithms are recently focusing on multi - core system to take full utilization of multiple processors available in the system. The design of parallel algorithm and performance measurement is the major issue on today's multi - core environment. Numerical problems arise in almost every branch of science which requires fast solution. In this paper we have presented parallel algorithms for computing the solution of system of non - linear equations and approximate the simple zeros of polynomial equations. The experimental results reveal that the performances of parallel algorithms are better than sequential. We implemented the parallel algorithms using multithreading features of OpenMP.

Software Design Challenges in Time Series Prediction Systems Using Parallel Implementation of Artificial Neural Networks.

Software development life cycle has been characterized by destructive disconnects between activities like planning, analysis, design, and programming. Particularly software developed with prediction based results is always a big challenge for designers. Time series data forecasting like currency exchange, stock prices, and weather report are some of the areas where an extensive research is going on for the last three decades. In the initial days, the problems with financial analysis and prediction were solved by statistical models and methods. For the last two decades, a large number of Artificial Neural Networks based learning models have been proposed to solve the problems of financial data and get accurate results in prediction of the future trends and prices. This paper addressed some architectural design related issues for performance improvement through vectorising the strengths of multivariate econometric time series models and Artificial Neural Networks. It provides an adaptive approach for predicting exchange rates and it can be called hybrid methodology for predicting exchange rates. This framework is tested for finding the accuracy and performance of parallel algorithms used.

Parallel Algorithm Performance Analysis and Optimization of Three-Dimensional Dendritic Growth Under Convection Based on OpenMP + LBM

Parallel Algorithm Performance Analysis and Optimization of Three-Dimensional Dendritic Growth Under Convection Based on OpenMP + LBM

Parallelization of Graph Isomorphism using OpenMP

in computer architecture leads to parallelize the sequential algorithm to exploit the concurrency provided by multiple core on single chip. Sequential programs do not gain performance from multicore. For multicore architectures, OPENMP and MPI are application programming interfaces. They can be used for parallelization of codes. For shared memory architecture OPENMP is used, whereas for distributed memory architecture MPI is used .In this paper, analysis of the performance of parallel algorithm over sequential algorithm is done. In particular, Graph Isomorphism problem based on vertex invariants is considered and parallelized using OpenMP. We demonstrate the performance of Graph Isomorphism using variable size graphs and parallelize it using vertical tiling technique on multicore architecture. Our previous work shows, sequential implementation of modified algorithm based on vertex invariants using Euclidian vector performs better than existing algorithm of Graph Isomorphism based on vertex invariants. To analyze the performance of parallel implementation, we present practical experiments with randomly generated graphs. KeywordsIsomorphism, Vertex Invariant, Euclidean vector,