Single Program Multiple Data Research Articles

A Distributed Block-Split Gibbs Sampler with Hypergraph Structure for High-Dimensional Inverse Problems

Sampling-based algorithms are classical approaches to perform Bayesian inference in inverse problems. They provide estimators with the associated credibility intervals to quantify the uncertainty on the estimators. Although these methods hardly scale to high dimensional problems, they have recently been paired with optimization techniques, such as proximal and splitting approaches, to address this issue. Such approaches pave the way to distributed samplers, splitting computations to make inference more scalable and faster. We introduce a distributed Split Gibbs sampler (SGS) to efficiently solve such problems involving distributions with multiple smooth and non-smooth functions composed with linear operators. The proposed approach leverages a recent approximate augmentation technique reminiscent of primal-dual optimization methods. It is further combined with a block-coordinate approach to split the primal and dual variables into blocks, leading to a distributed block-coordinate SGS. The resulting algorithm exploits the hypergraph structure of the involved linear operators to efficiently distribute the variables over multiple workers under controlled communication costs. It accommodates several distributed architectures, such as the Single Program Multiple Data and client-server architectures. Experiments on a large image deblurring problem show the performance of the proposed approach to produce high quality estimates with credibility intervals in a small amount of time. Supplementary material to reproduce the experiments is available online.

Performance Assessment of Using Docker for Selected MPI Applications in a Parallel Environment Based on Commodity Hardware

In the paper, we perform detailed performance analysis of three parallel MPI applications run in a parallel environment based on commodity hardware, using Docker and bare-metal configurations. The testbed applications are representative of the most typical parallel processing paradigms: master–slave, geometric Single Program Multiple Data (SPMD) as well as divide-and-conquer and feature characteristic computational and communication schemes. We perform analysis selecting best configurations considering various optimization flags for the applications and best execution times and speed-ups in terms of the number of nodes and overhead of the virtualized environment. We have concluded that for the configurations giving the shortest execution times the overheads of Docker versus bare-metal for the applications are as follows: 7.59% for master–slave run using 64 processes (number of physical cores), 15.30% for geometric SPMD run using 128 processes (number of logical cores) and 13.29% for divide-and-conquer run using 256 processes. Finally, we compare results obtained using gcc V9 and V7 compiler versions.

FEqa: Finite element computations on quantum annealers

The solution of physical problems discretized using the finite element methods using NISQ quantum hardware remains relatively unexplored. Here, we present a unified formulation (FEqa) to solve such problems using quantum annealers. FEqa is a hybrid technique in which the finite element problem is formulated on a classical computer, and the residual is minimized using a quantum annealer. The advantages of FEqa include utilizing a single qubit per degree of freedom, enforcing Dirichlet boundary conditions a priori, reaching arbitrary solution precision, and eliminating the possibility of the annealer generating invalid results. FEqa is scalable on the classical portion of the algorithm due to its Single Program Multiple Data (SPMD) nature and does not rely on ground state solutions from the annealer. The exponentially large number of collocation points used in quantum annealing are investigated for their cosine measures, and new iterative techniques are developed to exploit their properties. The quantum annealer has clear advantages in computational time over simulated annealing, for the example problems presented in this paper solved on the D-Wave machine. The presented work provides a pathway to solving physical problems using quantum annealers.

Numerical investigation of the Fourier–Kochin theory for wave-induced response estimation of floating structures

The Fourier–Kochin theory—an indirect solution approach for the boundary integral representations of free-surface flows that rely on free-surface Green function—is investigated by considering effective computation and reliable prediction. The present study addresses wave–body interaction with zero forward speed by adopting a higher-order approximation. Three distinct hull forms, the DTMB 5415, a scaled model reproduced from it, and a barge are used for practical application. The scaled frigate and barge models are used for comparative analysis to assess the performance of different computation techniques, modeling elements, and solution methods and to find the best course; the purpose of studying DTMB 5415 is to provide a realistic application based on a full-scale hull form by using the findings of the earlier benchmark studies. The analyses that cover both the rigid-body and elastic responses indicate that an adaptive approach for the computation of Fourier components has the potential to eliminate the numerical drawbacks of the presented implementation the Fourier–Kochin theory, but also stringently needs parallelism, preferably using the Single Program Multiple Data model. Using discontinuous elements for free-surface discretization when applying the Extended Boundary Integral Equation Method, the featured technique for irregular frequency suppression, is also promoted.

An Open-Source Many-Scenario Approach for Power System Dynamic Simulation on HPC Clusters

In this paper we introduce an approach to accelerate many-scenario (i.e., hundreds to thousands) power system simulations which is based on a highly scalable and flexible open-source software environment. In this approach, the parallel execution of simulations follows the single program, multiple data (SPMD) paradigm, where the dynamic simulation program is executed in parallel and takes different inputs to generate different scenarios. The power system is modeled using an existing Modelica library and compiled to a simulation executable using the OpenModelica Compiler. Furthermore, the parallel simulation is performed with the aid of a message-passing interface (MPI) and the approach includes dynamic workload balancing. Finally, benchmarks with the simulation environment are performed on high-performance computing (HPC) clusters with four test cases. The results show high scalability and a considerable parallel speedup of the proposed approach in the simulation of all scenarios.

Field Tests and Three-Dimensional Semi-Analytical Boundary Element Method Analysis of a Row of Holes as Active Barrier in Saturated Soil

Abstract The adverse effects of ground vibrations can be eliminated or prevented by installation of various types of wave barriers, such as a row of holes. The analysis is accomplished with the aid of a series of field experiments and numerical simulation. Three-dimensional semi-analytical boundary element models are established to use a row of holes as active wave barrier to isolate the ground vibrations generated by the machine foundation laid on the surface of saturated soil foundation, and the effects of the model parameters on isolation effectiveness are investigated and discussed in detail. A simple single program multiple data parallel algorithm is developed with the aid of Matlab software to increase calculating efficiency. The results show that a row of holes can isolate the ground vibrations successfully. Increasing the radius and the depth and decreasing the net spacing between two successive holes can all result in an increase in the screening effectiveness. According to the results, it is suggested in the design that the hole radius should take the value of 0.1∼0.15λR (λR is the Rayleigh wave length), and the hole depth and the net spacing should take the value no more than 1.0λR and 0.1λR, respectively. Moreover, the distance between the machine foundation and barrier can affect the screening effectiveness, and the larger the distance is, the poorer the screening effectiveness is. The number of holes in a row has less effect on the screening effectiveness. However, the bigger the number of holes in a row is, the larger the screening zone behind the barriers is.

History of coarrays and SPMD parallelism in Fortran

The coarray programming model is an expression of the Single-Program-Multiple-Data (SPMD) programming model through the simple device of adding a codimension to the Fortran language. A data object declared with a codimension is a coarray object. Codimensions express the idea that some objects are located in local memory while others are located in remote memory. Coarray syntax obeys most of the same rules for normal array syntax. It is familiar to the Fortran programmer so the use of coarray syntax is natural and intuitive. Although the basic idea is quite simple, inserting it into the language definition turned out to be difficult. In addition, the process was complicated by rapidly changing hardware and heated arguments over whether parallelism should be supported best as an interface to language-independent libraries, as a set of directives superimposed on languages, or as a set of specific extensions to existing languages. In this paper, we review both the early history of coarrays and also their development into a part of Fortran 2008 and eventually into a larger part of Fortran 2018. Coarrays have been used, for example, in weather forecasting and in neural networks and deep learning.

Compass SPMD: a SPMD vectorized tracking algorithm

Compass is a SPMD (Single Program Multiple Data) tracking algorithm for the upcoming LHCb upgrade in 2021. 40 Tb/s need to be processed in real-time to select events. Alternative frameworks, algorithms and architectures are being tested to cope with the deluge of data. Allen is a research and development project aiming to run the full HLT1 (High Level Trigger) on GPUs (Graphics Processing Units). Allen’s architecture focuses on data-oriented layout and algorithms to better exploit parallel architectures. GPUs already proved to exploit the framework efficiently with the algorithms developed for Allen, implemented and optimized for GPU architectures. We explore opportunities for the SIMD (Single Instruction Multiple Data) paradigm in CPUs through the Compass algorithm. We use the Intel SPMD Program Compiler (ISPC) to achieve good readability, maintainability and performance writing “GPU-like” source code, preserving the main design of the algorithm.

Race Condition Detection Algorithms

A data race is similar to any other bugs in software application. Data race will result in the execution of the program unpredictable. There are 46 documented races in Linux kernel. OpenMP is an Application programming interface for shared programming model. It is a construct based model which works on fork join parallelism. OpenMP achieved node level parallelism and can manage data in single instruction multiple data and single program multiple data parallelism by executing different constructs like work sharing and parallel constructs. In any shared programming model, variables are shared by multiple threads in the program to execute different tasks by different threads. OpenMP is used to achieve parallelism by creating shared variable environment but there are chances to have data races in OpenMP programs. In this paper we discuss different algorithms to detect data races in OpenMP programs.

VPPE: A Novel Visual Parallel Programming Environment

Parallel programming continues to be a challenging task despite the many advances in parallel architectures and their wide availability in the cloud. The need both to partition the workload among various processing elements and to specify communication between them to share code and data, and to coordinate their tasks, requires from the developer a deep understanding of the problem, the parallel architecture and the programming language used in order to develop efficient parallel applications. This problem can be reduced significantly through the use of visual programming languages to hide most aspects related to the specification of communication and processes management. This paper presents VPPE, a novel Visual Parallel Programming Environment that allows developers to program parallel applications through organising workflows of interconnected icons. VPPE is a cloud environment that supports icons for specifying: I/O operations, workflow organisation, communication, and processing. Processing computing patterns supported so far include Single Program Multiple Data, Multiple Program Multiple Data, Pipeline, and Master–Slave. The paper highlights the design of VPPE based on a context-free graph grammar, its current implementation based on Java-MPI, its use in developing various parallel applications, and its evaluation compared to Java-MPI text-based programming.

Parallel fast Fourier transform in SPMD style of Cilk

In this paper, we propose a parallel one-dimensional non-recursive fast Fourier transform (FFT) program based on conventional Cooley-Tukey's algorithm written in C using Cilk in single program multiple data (SPMD) style. As a highly compact designed code, this code is compared with a highly tuned parallel recursive fast Fourier transform (FFT) using Cilk, which is included in Cilk package of version 5.4.6. Both algorithms are executed on multicore servers, and experimental results show that the performance of the SPMD style of Cilk fast Fourier transform (FFT) parallel code is highly competitive and promising.

Prediction of Energy Consumption by Checkpoint/Restart in HPC

The fault tolerance method most used today in high-performance computing (HPC) is coordinated checkpointing. This, like any other fault tolerance method, adds additional energy consumption to that of the execution of the application. Currently, knowing and minimizing this energy consumption is a challenge. The objective of this paper is to propose a model to estimate the energy consumption of checkpoint and restart operations and a method for its construction. These estimates allow the evaluation of different scenarios in order to minimize energy consumption. We focus on coordinated checkpoint/restart at the system level, in single-program multiple-data (SPMD) applications, on homogeneous clusters. We study the behavior of the power dissipated by the compute node during a checkpoint/restart operation, as well as its execution time, considering different parameters of the system and the application. The experimentation carried out on two platforms shows the validity of the proposal. We also evaluate the impact on power and energy consumption of the processor’s C states, the configuration of the network file system (NFS), where the checkpoint files are stored, and the compression of the checkpoint files. This paper contributes to the objective of predicting energy consumption in the execution of applications that use checkpoint/restart. Not counting the outliers, we can estimate the energy consumed by checkpoint/restart operations with errors lower than 7.5%.

Local rollback for resilient MPI applications with application-level checkpointing and message logging

The resilience approach generally used in high-performance computing (HPC) relies on coordinated checkpoint/restart, a global rollback of all the processes that are running the application. However, in many instances, the failure has a more localized scope and its impact is usually restricted to a subset of the resources being used. Thus, a global rollback would result in unnecessary overhead and energy consumption, since all processes, including those unaffected by the failure, discard their state and roll back to the last checkpoint to repeat computations that were already done. The User Level Failure Mitigation (ULFM) interface – the last proposal for the inclusion of resilience features in the Message Passing Interface (MPI) standard – enables the deployment of more flexible recovery strategies, including localized recovery. This work proposes a local rollback approach that can be generally applied to Single Program, Multiple Data (SPMD) applications by combining ULFM, the ComPiler for Portable Checkpointing (CPPC) tool, and the Open MPI VProtocol system-level message logging component. Only failed processes are recovered from the last checkpoint, while consistency before further progress in the execution is achieved through a two-level message logging process. To further optimize this approach point-to-point communications are logged by the Open MPI VProtocol component, while collective communications are optimally logged at the application level—thereby decoupling the logging protocol from the particular collective implementation. This spatially coordinated protocol applied by CPPC reduces the log size, the log memory requirements and overall the resilience impact on the applications.

The electromagnetic waves propagation in unmagnetized plasma media using parallelized finite-difference time-domain method

The finite-difference time-domain (FDTD) method has been commonly utilized to simulate the electromagnetic (EM) waves propagation in the plasma media. However, the FDTD method may bring about extra run-time on concerning computationally large and complicated EM problems. Fortunately, the FDTD method is easy to parallelize. Besides, GPU has been widely used for parallel computing due to its unique SPMD (Single Program Multiple Data) architecture. In this paper, we represent the parallel Runge-Kutta exponential time differencing scheme FDTD (RKETD) method for the unmagnetized plasma implemented on GPU. The detailed flowchart of parallel RKETD-FDTD method is described. The accuracy and acceleration performance of the proposed parallel RKETD-FDTD method implemented on GPU are substantiated by calculating the reflection and transmission coefficients for one-dimensional unmagnetized plasma slab. The results indicate that the numerical precision of the parallel RKETD-FDTD scheme is consistent with that of the code implemented on CPU. The computation efficiency is greatly improved compared with merely CPU-based serial RKETD-FDTD method. Moreover, the comparisons of the performance of CUDA-based GPU parallel program, Open MP (Open Multi-Processing)-based CPU parallel program, and single-CPU serial program on the same host computer are done. Compared with the serial program, both parallel programs get good results, while GPU-based parallel program gains better result.

Teaching distributed memory programming from mental models

Distributed memory programming, typically through the MPI library, is the de facto standard for programming large scale parallelism, with up to millions of individual processes. Its dominant paradigm of Single Program Multiple Data (SPMD) programming is different from threaded and multicore parallelism, to an extent that students have a hard time switching models. In contrast to threaded programming, which allows for a view of the execution with central control and a central repository of data, SPMD programming has a symmetric model where all processes are active all the time, with none privileged, and where data is distributed.This model is counterintuitive to the novice parallel programmer, so care needs to be taken how to instill the proper ‘mental model’. Adoption of an incorrect mental model leads to broken or inefficient code.We identify problems with the currently common way of teaching MPI, and propose a structuring of MPI courses that is geared to explicit reinforcing the symmetric model. Additionally, we advocate starting from realistic scenarios, rather than writing artificial code just to exercise newly-learned routines.

Parallel fast Fourier transform in SPMD style of CILK

In this paper, we propose a parallel one-dimensional non-recursive fast Fourier transform (FFT) program based on conventional Cooley-Tukey's algorithm written in C using Cilk in single program multiple data (SPMD) style. As a highly compact designed code, this code is compared with a highly tuned parallel recursive fast Fourier transform (FFT) using Cilk, which is included in Cilk package of version 5.4.6. Both algorithms are executed on multicore servers, and experimental results show that the performance of the SPMD style of Cilk fast Fourier transform (FFT) parallel code is highly competitive and promising.

DITVA: Dynamic Inter-Thread Vectorization Architecture

In the Single-Program Multiple-Data (SPMD) programming model, threads of an application exhibit very similar control flows and often execute the same instructions, but on different data. In this paper, we propose the Dynamic Inter-thread Vectorization Architecture (DITVA) to leverage the implicit Data Level Parallelism that exists across threads on SPMD applications.By assembling dynamic vector instructions at runtime, DITVA extends an in-order SMT processor with a dynamic inter-thread vector execution mode akin to the Single-Instruction, Multiple-Thread model of Graphics Processing Units. In this mode, multiple scalar threads running in lockstep share a single instruction stream and their respective instruction instances are aggregated into SIMD instructions. DITVA can leverage existing SIMD units and maintains binary compatibility with existing CPU architectures. To balance thread- and data-level parallelism, threads are statically grouped into fixed-size independently scheduled warps. Additionally, to maximize dynamic vectorization opportunities, we adapt the fetch steering policy to favor thread synchronization within warps and thus improve lockstep execution.Our experimental evaluation of the DITVA architecture on the SPMD applications from the PARSEC and Rodinia OpenMP benchmarks show that a 4-warp × 4-lane 4-issue DITVA architecture with a realistic bank-interleaved cache achieves 1.55× higher performance compared to a 4-thread 4-issue SMT architecture with AVX instructions, while fetching and issuing 51% fewer instructions, and achieving an overall 24% energy reduction. DITVA also enables applications limited by memory to scale with higher bandwidth architectures. For instance, when the bandwidth is increased from 2GB/s to 16GB/s, we find that memory bound applications show an improvement in performance by 3× in comparison with the baseline SMT. Therefore, DITVA appears as a cost-effective design for achieving very high single-core performance on SPMD parallel sections.

DGR-ELM–Distributed Generalized Regularized ELM for classification

Abstract Extreme Learning Machine (ELM) has recently increased popularity and has been successfully applied to a wide range of applications. Variants using regularization are now a common practice in the state of the art in ELM field. The most commonly used regularization is the l2 norm, which improves generalization but result in a dense network. Regularization based on the elastic net has also been proposed but mainly applied to regression and binary classification problems. In this paper, we propose a generalization of regularized ELM (R-ELM) for multiclass classification problems, termed GR-ELM. We achieve such generalization using the l2,1 and Frobenius norm. Traditional R-ELM is a particular case of our method when binary classification tasks are considered. We also propose an alternative algorithm for GR-ELM when training data is distributed, namely DGR-ELM. We use Alternating Direction Method of Multipliers (ADMM) for solving the resulting optimization problems. Message Passing Interface (MPI) in a Single Program, Multiple Data (SPMD) programming style is chosen for implementing DGR-ELM. Extensive experiments are conducted to evaluate the proposed method. Our experiments show that GR-ELM and DGR-ELM have similar training and testing accuracy when compared to R-ELM, although usually faster testing time is obtained with our method due to the compactness of the resulting network.

A technique to automatically determine Ad-hoc communication patterns at runtime

Abstract Current High Performance Computing (HPC) systems are typically built as interconnected clusters of shared-memory multicore computers. Several techniques to automatically generate parallel programs from high-level parallel languages or sequential codes have been proposed. To properly exploit the scalability of HPC clusters, these techniques should take into account the combination of data communication across distributed memory, and the exploitation of shared-memory models. In this paper, we present a new communication calculation technique to be applied across different SPMD (Single Program Multiple Data) code blocks, containing several uniform data access expressions. We have implemented this technique in Trasgo, a programming model and compilation framework that transforms parallel programs from a high-level parallel specification that deals with parallelism in a unified, abstract, and portable way. The proposed technique computes at runtime exact coarse-grained communications for distributed message-passing processes. Applying this technique at runtime has the advantage of being independent of compile-time decisions, such as the tile size chosen for each process. Our approach allows the automatic generation of pre-compiled multi-level parallel routines, libraries, or programs that can adapt their communication, synchronization, and optimization structures to the target system, even when computing nodes have different capabilities. Our experimental results show that, despite our runtime calculation, our approach can automatically produce efficient programs compared with MPI reference codes, and with codes generated with auto-parallelizing compilers.

Large Scale Document Inversion using a Multi-threaded Computing System.

•Information systems➝Information retrieval • Computing methodologies➝Massively parallel and high-performance simulations.