Unified Parallel Research Articles

A Unified Parallel Framework for LUT Mapping and Logic Optimization

A Unified Parallel Framework for LUT Mapping and Logic Optimization

A Unified Parallel CORDIC-Based Hardware Architecture for LSTM Network Acceleration

Deep Neural Networks (DNNs) have recently become the standard tool for solving various practical problems in a wide range of applications with state-of-the-art performance. Recurrent Neural Networks (RNNs) such as Long Short-Term Memory (LSTM) are a subset of DNNs with fully connected single or multi-layer networks. The complex neurons and internal states of LSTM networks enable them to build a memory of events, making them ideal for time series applications. Despite the great potential of LSTM networks, their heterogeneous operations and computational resource requirements create a vast gap when it comes to the fast processing time required in real-time applications using low-power, low-cost edge devices. This work proposes a novel hardware architecture that combines serial-parallel computation with matrix algebra concepts and efficient low-power computer arithmetics for LSTM network acceleration. The hardware is based on a systolic ring of outer-product-based processing elements (PEs) and a reusable single activation function block (AFB). PEs and AFB are implemented using the coordinate rotation digital computer algorithm (CORDIC) in the linear and hyperbolic modes. Unlike most approaches, the proposed hardware can be configured to perform recurrent and non-recurrent fully connected layers (FC) computations, making it suitable for various low-power edge applications. The architecture is validated on the Xilinx PYNQ-Z1 development board using an open-source time series dataset. The implemented design achieves <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$114 \mu s$</tex-math></inline-formula> average latency and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$1.8GOPS$</tex-math></inline-formula> throughput. The proposed design's low latency and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$0.438W$</tex-math></inline-formula> power consumption makes it suitable for resource-constrained edge platforms.

Multiscale homogenized predictive modelling of flooding surface in urban cities using physics-induced deep AI with UPC

Climate change is expected to worsen streamflow conditions, in terms of their frequency and magnitude, in urbanized watersheds, which can directly result in flood scenarios of greater water surface displacements. To contribute to the scientific community's present flood mitigation and assessment activities, this study develops a generic engineering approach to construct multiscale homogenized deep neural networks (MHDNN), implemented with unified parallel C (MHDNN-UPC), by fusing mass and momentum conservation laws, as part of hydrodynamic modelling, with deep learning (DL) computations for the predictive modelling of water surface displacements due to peak streamflow conditions, as representative of flood scenarios. The approach is comprised of a series of systematic analyses, namely: (Phase A) derive homogenized effective solutions, via homogenization theory coupled with multiscale perturbation analysis for hydrodynamic modelling, to perform features extractions and constructing useful activation functions for training MHDNN predictive model(s); (Phase B) UPC implementation of MHDNN model(s) to improve their computational performance; and (Phase C) using available field datasets, pertaining to surface displacements in watersheds, to train, validate and test MHDNN-UPC model(s). The proposed predictive approach is then verified across 48 selected states in the United States for modelling their recorded displacements over the past 100 years, by achieving an average 10% improvement in its predictive accuracy, as compared to other traditional models, while also improving the model's computational performance with the implemented UPC component.

Advanced Parallelism of DGTD Method With Local Time Stepping Based on Novel MPI + MPI Unified Parallel Algorithm

In this communication, a novel message passing interface (MPI) parallel algorithm for nodal discontinuous Galerkin time-domain (NDGTD) method has been developed. A unified MPI + MPI technique has been introduced for extreme parallelism on a large-scale computer cluster. Through the data transmission between CPU nodes using MPI persistent nonblocking two-side communication and the direct data connection between processors in the same node via MPI shared memory windows, a two-layered parallel architecture is implemented to minimize the communication. To further accelerate the solution of the multiscale problems, the local time stepping (LTS) technique has been employed in the NDGTD method. A fast time step estimation method has been presented in this communication. With high overlap between the information transmission and the data calculation, the proposed MPI + MPI scheme overcomes the degradation of the parallel efficiency of the pure MPI technique in the scenario of the LTS technique and the large-scale CPU cores. Up to 94% parallel efficiency in 6400 CPU cores is achieved for the average single-core loading about 1700 finite elements, and 18 times acceleration for time step estimation can be obtained with the fourth-order basis function. Three practical complex examples are given to demonstrate a good performance of the proposed method.

A unified cloud-enabled discrete event parallel and distributed simulation architecture

Cloud infrastructure provides rapid resource provision for on-demand computational requirements. Cloud simulation environments today are largely employed to model and simulate complex systems for remote accessibility and variable capacity requirements. In this regard, scalability issues in Modeling and Simulation (M&S) computational requirements can be tackled through the elasticity of on-demand Cloud deployment. However, implementing a high performance cloud M&S framework following these elastic principles is not a trivial task as parallelizing and distributing existing architectures is challenging. Indeed, both the parallel and distributed M&S developments have evolved following separate ways. Parallel solutions has always been focused on ad-hoc solutions, while distributed approaches, on the other hand, have led to the definition of standard distributed frameworks like the High Level Architecture (HLA) or influenced the use of distributed technologies like the Message Passing Interface (MPI). Only a few developments have been able to evolve with the current resilience of computing hardware resources deployment, largely focused on the implementation of Simulation as a Service (SaaS), albeit independently of the parallel ad-hoc methods branch. In this paper, we present a unified parallel and distributed M&S architecture with enough flexibility to deploy parallel and distributed simulations in the Cloud with a low effort, without modifying the underlying model source code, and reaching important speedups against the sequential simulation, especially in the parallel implementation. Our framework is based on the Discrete Event System Specification (DEVS) formalism. The performance of the parallel and distributed framework is tested using the xDEVS M&S tool, Application Programming Interface (API) and the DEVStone benchmark with up to eight computing nodes, obtaining maximum speedups of 15.95× and 1.84×, respectively.

Nonlocal weighted sparse unmixing based on global search and parallel optimization

Sparse unmixing (SU) can represent an observed image using pure spectral signatures and corresponding fractional abundance from a large spectral library and is an important technique in hyperspectral unmixing. However, the existing SU algorithms mainly exploit spatial information from a fixed neighborhood system, which is not sufficient. To solve this problem, we propose a nonlocal weighted SU algorithm based on global search (G-NLWSU). By exploring the nonlocal similarity of the hyperspectral image, the weights of pixels are calculated to form a matrix to weight the abundance matrix. Specifically, G-NLWSU first searches for a similar group of each pixel in the global scope then uses singular value decomposition to denoise and finally obtains the weight matrix by considering correlations between similar pixels. To reduce the execution burden of G-NLWSU, we propose a parallel computing version of G-NLWSU, named PG-NLWSU, which integrates compute unified device architecture-based parallel computing into G-NLWSU to accelerate the search for groups of nonlocally similar pixels. Our proposed algorithms shed new light on SU by considering a new exploitation process of spatial information and parallel computing scenario. Experimental results conducted on simulated and real datasets show that PG-NLWSU is superior to comparison algorithms.

Oblivious Resampling Oracles and Parallel Algorithms for the Lopsided Lovász Local Lemma

The Lovász Local Lemma (LLL) shows that, for a collection of “bad” events B in a probability space that are not too likely and not too interdependent, there is a positive probability that no events in B occur. Moser and Tardos (2010) gave sequential and parallel algorithms that transformed most applications of the variable-assignment LLL into efficient algorithms. A framework of Harvey and Vondrák (2015) based on “resampling oracles” extended this to sequential algorithms for other probability spaces satisfying a generalization of the LLL known as the Lopsided Lovász Local Lemma (LLLL). We describe a new structural property that holds for most known resampling oracles, which we call “obliviousness.” Essentially, it means that the interaction between two bad-events B , B ′ depends only on the randomness used to resample B and not the precise state within B itself. This property has two major consequences. First, combined with a framework of Kolmogorov (2016), it leads to a unified parallel LLLL algorithm, which is faster than previous, problem-specific algorithms of Harris (2016) for the variable-assignment LLLL and of Harris and Srinivasan (2014) for permutations. This gives the first RNC algorithms for rainbow perfect matchings and rainbow Hamiltonian cycles of K n . Second, this property allows us to build LLLL probability spaces from simpler “atomic” events. This gives the first resampling oracle for rainbow perfect matchings on the complete s -uniform hypergraph K n ( s ) and the first commutative resampling oracle for Hamiltonian cycles of K n .

UNIFIED PARALLEL ALGORITHM AND PROGRAMMING COMPLEX OF OPTIMAL PLANNING OF NON-UNIFORM FLOWS IN THE NETWORKS

Purpose. The purpose of the article is to develop a universal unified parallel synchronous algorithm for the implementation of tasks for calculation of maximum one- and multicommodity flows, as well as the creation of a software complex that provides the formation of surface graph models of flows and performs optimal planning of non-uniform flows in transport and other networks. Methodology. The paper investigates the possibilities of previously created and comprehensively verified heuristic parallel synchronous algorithm for calculating maximum one- and multicommodity flows in the networks, establishes its potential limitations, and determines additional advanced procedures that transform the algorithm into a universal parallel algorithm. The proposed parallel synchronous algorithm uses a width-first search strategy while simultaneously identifying possible paths of flows through the network with an estimation of their throughput. Herewith the possibility of analyzing several incremental flows across the network in one iteration was studied. Findings. The article proposes a universal unified parallel synchronous algorithm for calculating maximum flows in networks and develops a unified procedure and software package for planning of non-uniform as well as competitive flows in transport and other networks. The developed software complex implements the problems of formation of surface graph models of networks, for which the problem of optimal planning of non-uniform and competitive multicriteria flows in transport networks is solved. Originality. The article develops a new universal unified parallel synchronous algorithm and procedure for the calculation of optimal uniform, multicommodity and competitive flows in transport networks. Practical value. The practical value of the obtained results is determined by the universal capabilities and efficiency of the procedure for planning non-uniform flows in the networks based on the application of a new parallel synchronous algorithm, as well as the developed software complex, which provides the ability to solve the problems of analysis and planning of uniform and multicommodity flows in transport networks, as well as the implementation of calculation tasks of competitive models of transport and information flows formation. The software complex has a built-in editor of interactive network modeling and a toolbar, which provides both creation of new and downloading existing graphs of networks from the modeling libraries, preservation of optimum flows in the network in the form of an image and a text file, output of errors when working with the program.

Optimizing Speedup Performance of Computational Hydrodynamic Simulations with UPC Programming Model

AbstractIn this study, we exploit the advantages of Berkeley’s Unified Parallel C (UPC) programming model to optimize the speedup performance of computational hydrodynamic (CHD) simulations, which ...

Ray-based modeling and imaging in viscoelastic media using graphics processing units

In seismic data processing, the amplitude loss caused by attenuation should be taken into account. The basis for this is provided by a 3D attenuation model described by the quality factor [Formula: see text], which is used in viscoelastic modeling and imaging. We have accomplished viscoelastic modeling and imaging using ray theory and the ray-Born approximation. This makes it possible to take [Formula: see text] into account using complex-valued and frequency-dependent traveltimes. We have developed a unified parallel implementation for modeling and imaging in the frequency domain and carried out the numerical integration on a graphics processing unit. A central part of the implementation is an efficient technique for computing large integrals. We applied the integration method to the 3D SEG/EAGE overthrust model to generate synthetic seismograms and imaging results. The attenuation effects are accurately modeled in the seismograms and compensated for in the imaging algorithm. The results indicate a significant improvement in computational efficiency compared to a parallel central processing unit baseline.

Performance Optimization and Modeling of Fine-Grained Irregular Communication in UPC

The Unified Parallel C (UPC) programming language offers parallelism via logically partitioned shared memory, which typically spans physically disjoint memory subsystems. One convenient feature of UPC is its ability to automatically execute between-thread data movement, such that the entire content of a shared data array appears to be freely accessible by all the threads. The programmer friendliness, however, can come at the cost of substantial performance penalties. This is especially true when indirectly indexing the elements of a shared array, for which the induced between-thread data communication can be irregular and have a fine-grained pattern. In this paper, we study performance enhancement strategies specifically targeting such fine-grained irregular communication in UPC. Starting from explicit thread privatization, continuing with block-wise communication, and arriving at message condensing and consolidation, we obtained considerable performance improvement of UPC programs that originally require fine-grained irregular communication. Besides the performance enhancement strategies, the main contribution of the present paper is to propose performance models for the different scenarios, in the form of quantifiable formulas that hinge on the actual volumes of various data movements plus a small number of easily obtainable hardware characteristic parameters. These performance models help to verify the enhancements obtained, while also providing insightful predictions of similar parallel implementations, not limited to UPC, that also involve between-thread or between-process irregular communication. As a further validation, we also apply our performance modeling methodology and hardware characteristic parameters to an existing UPC code for solving a 2D heat equation on a uniform mesh.

PLS for Big Data: A unified parallel algorithm for regularised group PLS

Partial Least Squares (PLS) methods have been heavily exploited to analyse the association between two blocks of data. These powerful approaches can be applied to data sets where the number of variables is greater than the number of observations and in the presence of high collinearity between variables. Different sparse versions of PLS have been developed to integrate multiple data sets while simultaneously selecting the contributing variables. Sparse modeling is a key factor in obtaining better estimators and identifying associations between multiple data sets. The cornerstone of the sparse PLS methods is the link between the singular value decomposition (SVD) of a matrix (constructed from deflated versions of the original data) and least squares minimization in linear regression. We review four popular PLS methods for two blocks of data. A unified algorithm is proposed to perform all four types of PLS including their regularised versions. We present various approaches to decrease the computation time and show how the whole procedure can be scalable to big data sets. The bigsgPLS R package implements our unified algorithm and is available at https://github.com/matt-sutton/bigsgPLS.

Modelling of Communication in Unified Parallel and Distributed Computing Environment

In modelling of sequential computers and algorithms, communication delay has often been ignored. Later, in modelling of parallel computers and parallel algorithms based on shared memory, it was supposed that the influence of communication is lower compared to computation complexity. It was supposed that, in better cases, communication delay is included in analysed computation complexity. However, a different situation can occur in high performance processors (CPU) or simplified CPUs called cores within the parallel computing node (multiprocessor/ multicore SMP). The similar case could occur in the multiple use of high performing computing nodes as the building computing nodes of modern parallel computers NOW (network of workstations) or Grid module (network of NOW modules). However, if complex problem are intended to be solved in a parallel way (parallel algorithm), they generate additional communication steps which correspond to inter process communication (IPC) among the decomposed parts (parallel processes) of the given solved problem. In general, it can be said that any parallel algorithm (PA) consists of created parallel processes having sequential character and IPC communication. Therefore, the increased role of IPC communication has been analysed in this paper through the analytical modelling of IPC complexity (number of communication steps) in a similar way as the one used to analyse computation complexity in sequential algorithms or parallel processes). Based on the defined communication parameters and the extensions of the complexity theory to PA, a unique modelling approach was developed to model the communication complexity of all existing parallel computers and PA. The secondary problem is to model critical parts of IPC influence for PA performance and that, at first, in minimisation of IPC steps of decomposed parallel processes and secondly at whole performance optimisation of the PA. The achieved results showed, from the user’s point of view, the important influence of modelling IPC complexity in PA on the illustrated real examples.

Evaluating the networking characteristics of the Cray XC‐40 Intel Knights Landing‐based Cori supercomputer at NERSC

SummaryThere are many potential issues associated with deploying the Intel Xeon PhiTM (code named Knights Landing [KNL]) manycore processor in a large‐scale supercomputer. One in particular is the ability to fully utilize the high‐speed communications network, given that the serial performance of a Xeon PhiTM core is a fraction of a Xeon®core. In this paper, we take a look at the trade‐offs associated with allocating enough cores to fully utilize the Aries high‐speed network versus cores dedicated to computation, eg, the trade‐off between MPI and OpenMP. In addition, we evaluate new features of Cray MPI in support of KNL, such as internode optimizations. We also evaluate one‐sided programming models such as Unified Parallel C. We quantify the impact of the above trade‐offs and features using a suite of National Energy Research Scientific Computing Center applications.

Performance Evaluation of a Two-Dimensional Lattice Boltzmann Solver Using CUDA and PGAS UPC Based Parallelisation

The Unified Parallel C (UPC) language from the Partitioned Global Address Space (PGAS) family unifies the advantages of shared and local memory spaces and offers a relatively straightforward code parallelisation with the Central Processing Unit (CPU). In contrast, the Computer Unified Device Architecture (CUDA) development kit gives a tool to make use of the Graphics Processing Unit (GPU). We provide a detailed comparison between these novel techniques through the parallelisation of a two-dimensional lattice Boltzmann method based fluid flow solver. Our comparison between the CUDA and UPC parallelisation takes into account the required conceptual effort, the performance gain, and the limitations of the approaches from the application oriented developers’ point of view. We demonstrated that UPC led to competitive efficiency with the local memory implementation. However, the performance of the shared memory code fell behind our expectations, and we concluded that the investigated UPC compilers could not efficiently treat the shared memory space. The CUDA implementation proved to be more complex compared to the UPC approach mainly because of the complicated memory structure of the graphics card which also makes GPUs suitable for the parallelisation of the lattice Boltzmann method.

Indexed-Points Parallel Coordinates Visualization of Multivariate Correlations.

We address the problem of visualizing multivariate correlations in parallel coordinates. We focus on multivariate correlation in the form of linear relationships between multiple variables. Traditional parallel coordinates are well prepared to show negative correlations between two attributes by distinct visual patterns. However, it is difficult to recognize positive correlations in parallel coordinates. Furthermore, there is no support to highlight multivariate correlations in parallel coordinates. In this paper, we exploit the indexed point representation of p -flats (planes in multidimensional data) to visualize local multivariate correlations in parallel coordinates. Our method yields clear visual signatures for negative and positive correlations alike, and it supports large datasets. All information is shown in a unified parallel coordinates framework, which leads to easy and familiar user interactions for analysts who have experience with traditional parallel coordinates. The usefulness of our method is demonstrated through examples of typical multidimensional datasets.

Evaluation of the partitioned global address space (PGAS) model for an inviscid Euler solver

In this paper we evaluate the performance of Unified Parallel C (which implements the partitioned global address space programming model) using a numerical method that is widely used in fluid dynamics. In order to evaluate the incremental approach to parallelization (which is possible with UPC) and its performance characteristics, we implement different levels of optimization of the UPC code and compare it with an MPI parallelization on four different clusters of the Austrian HPC infrastructure (LEO3, LEO3E, VSC2, VSC3) and on an Intel Xeon Phi. We find that UPC is significantly easier to develop in compared to MPI and that the performance achieved is comparable to MPI in most situations. The obtained results show worse performance (on VSC2), competitive performance (on LEO3, LEO3E and VSC3), and superior performance (on the Intel Xeon Phi).

EM scattering calculation of large sea surface with SSA method at S, X, Ku, and K bands

The practical method, small slope approximation is employed to analyze EM scattering from sea surface. Considering the simulation of EM scattering from actual scene, the size of sea surface is very large and it should be much larger than the target’s size if a target exists. However, the memory consumption is huge and always larger than the existing memory of personal computer (PC) when EM frequency is at X band or higher band. Here, the spectral decomposition modeling (SDM) is proposed to decrease memory consumption without reducing efficiency. The SDM method uses quite less memory to keep the data of large sea surface, so that the memory insufficient problem can be avoided. The scattering coefficients and Doppler spectra of the sea surfaces generated by the proposed method fit well with the results calculated by general method when interpolation method with four points is applied. Moreover, higher efficiency can be obtained by applying the compute unified device architecture-based parallel method for the SDM method involves less data to transmit than the general method.

A general model checking framework for various memory consistency models

Relaxed memory consistency models are common and essential when multiple processes share a single global address space, such as when using multicore CPUs, distributed shared-memory programming languages, and so forth. Programming within these models is difficult and error prone, because of non-intuitive behaviors that could not occur in a strict consistency model. In addition, because the memory consistency models vary from language to language, and CPU to CPU, a program that may work correctly on one system may not work on another. To address the problem, this paper describes a model checking framework in which users are able to check their programs under various memory consistency models. More specifically, our framework provides a base model that exhibits very relaxed behaviors, and users are able to define various consistency models by adding constraints to the base model. This paper also describes McSPIN, a prototype implementation of a model checker based on the proposed framework. McSPIN can take a memory consistency model as an input, as well as a program and a property to be checked. We have specified the necessary constraints for three practical existing memory consistency models (Unified Parallel C, Coarray Fortran, and Itanium). McSPIN verified some example programs correctly, and confirmed the expected differences among the three models.

Using shared-data localization to reduce the cost of inspector-execution in unified-parallel-C programs

We improve performance of fine-grain UPC applications by orders of magnitude.We introduce a novel shared-data localization transformation.We present a thorough performance analysis and evaluation.We show that reducing run-time calls is crucial for performance.We achieve performance comparable to C and MPI using the UPC programming model. Programs written in the Unified Parallel C (UPC) language can access any location of the entire local and remote address space via read/write operations. However, UPC programs that contain fine-grained shared accesses can exhibit performance degradation. One solution is to use the inspector-executor technique to coalesce fine-grained shared accesses to larger remote access operations. A straightforward implementation of the inspector-executor transformation results in excessive instrumentation that hinders performance.This paper addresses this issue and introduces various techniques that aim at reducing the generated instrumentation code: a shared-data localization transformation based on Constant-Stride Linear Memory Descriptors (CSLMADs) S. Aarseth, Gravitational N-Body Simulations: Tools and Algorithms, Cambridge Monographs on Mathematical Physics, Cambridge University Press, 2003., the inlining of data locality checks and the usage of an index vector to aggregate the data. Finally, the paper introduces a lightweight loop code motion transformation to privatize shared scalars that were propagated through the loop body.A performance evaluation, using up to 2048 cores of a POWER 775, explores the impact of each optimization and characterizes the overheads of UPC programs. It also shows that the presented optimizations increase performance of UPC programs up to 1.8 × their UPC hand-optimized counterpart for applications with regular accesses and up to 6.3 × for applications with irregular accesses.