Shared Memory Programming Model Research Articles

Evaluating performance portability of five shared-memory programming models using a high-order unstructured CFD solver

This paper presents implementing and optimizing a high-order unstructured computational fluid dynamics (CFD) solver using five shared-memory programming models: CUDA, OpenACC, OpenMP, Kokkos, and OP2. The study aims to evaluate the performance of these models on different hardware architectures, including NVIDIA GPUs, x86-based Intel/AMD, and Arm-based systems. The goal is to determine whether these models can provide developers with performance-portable solvers running efficiently on various architectures. The paper forms a more holistic view of a high-order solver across multiple platforms by visualizing performance portability (PP) and measuring productivity. It gives guidelines for translating existing codebases and their data structures to these models.

연산 속도 향상을 위한 병렬 연산 기반 상태 추정

Since state estimation is a real-time application program, its computational speed is important as well as accuracy. Therefore, improving the computation speed of state estimation is a very significant issue. This paper studied the method of improving computational speed of state estimation by using the parallel computing technique. In order to apply the parallel computing technique to state estimation, OpenMP was used as a programming tool. This tool is a shared memory programming model that requires no other devices such as GPU(graphics processing unit) and DSP(digital signal processor), and has the advantage of the fastest data transfer rate between memories. This paper proposes an algorithm that divides the entire system into small scales and performs state estimation in parallel with the divided systems. The proposed algorithm can improve the speed of computing state estimation and also the performance of the bad data processing that may happen for some reasons.

Collectives in hybrid MPI+MPI code: Design, practice and performance

The use of hybrid scheme combining the message passing programming models for inter-node parallelism and the shared memory programming models for node-level parallelism is widely spread. Existing extensive practices on hybrid Message Passing Interface (MPI) plus Open Multi-Processing (OpenMP) programming account for its popularity. Nevertheless, strong programming efforts are required to gain performance benefits from the MPI+OpenMP code. An emerging hybrid method that combines MPI and the MPI shared memory model (MPI+MPI) is promising. However, writing an efficient hybrid MPI+MPI program – especially when the collective communication operations are involved – is not to be taken for granted.In this paper, we propose a new design method to implement hybrid MPI+MPI context-based collective communication operations. Our method avoids on-node memory replications (on-node communication overheads) that are required by semantics in pure MPI. We also offer wrapper primitives hiding all the design details from users, which comes with practices on how to structure hybrid MPI+MPI code with these primitives. Further, the on-node synchronization scheme required by our method/collectives gets optimized. The micro-benchmarks show that our collectives are comparable or superior to those in pure MPI context. We have further validated the effectiveness of the hybrid MPI+MPI model (which uses our wrapper primitives) in three computational kernels, by comparison to the pure MPI and hybrid MPI+OpenMP models.

The Method of Improving the Performance of Network Analysis Application for the Whole Power Grid

With the rapid development of UHV AC / DC hybrid power grid, it is required that the Network Analysis Application have the ability of unified analysis and high-performance computing. In this paper, the time-consuming analysis of each sub-function of Network Analysis Application is carried out to realize the performance bottleneck analysis of the whole process of network analysis. It is proposed that the performance of network analysis should be improved in three aspects: data input and output, topology analysis and core computing of Network Analysis Application. In the aspect of data input, the grid model data service is constructed to realize the combination of measurement reading and model reading, and the parallel model verification and boundary equivalence are completed; In the aspect of data output, it can be parallelized by table, block and equipment; In the aspect of topology analysis, the shared memory programming model OpenMP is adopted, and based on the fork/join parallel mode, its parallelization is realized; In the aspect of core computing, the existing parallel computing methods are summarized, and through the actual power grid simulation analysis, it puts forward the parallel computing mode applicable to different scale power grids and different applications. Finally, the effectiveness of this method is verified by comparing the optimized performance of state estimation.

AllScale API

Effectively implementing scientific algorithms in distributed memory parallel applications is a difficult task for domain scientists, as evident by the large number of domain-specific languages and libraries available today attempting to facilitate the process. However, they usually provide a closed set of parallel patterns and are not open for extension without vast modifications to the underlying system. In this work, we present the AllScale API, a programming interface for developing distributed memory parallel applications with the ease of shared memory programming models. The AllScale API is closed for a modification but open for an extension, allowing new user-defined parallel patterns and data structures to be implemented based on existing core primitives and therefore fully supported in the AllScale framework. Focusing on high-level functionality directly offered to application developers, we present the design advantages of such an API design, detail some of its specifications and evaluate it using three real-world use cases. Our results show that AllScale decreases the complexity of implementing scientific applications for distributed memory while attaining comparable or higher performance compared to MPI reference implementations.

HDOT — An approach towards productive programming of hybrid applications

A wealth of important scientific and engineering applications are configured for use on high performance computing architectures using functionality found in the MPI specification. This specification provides application developers with a straightforward means for implementing their ideas for execution on distributed-memory parallel processing computers. OpenMP directives provide a means for operating on shared-memory regions of those computers. With the advent of machines composed of many-core processors, the strict synchronisation required by the bulk synchronous parallel (BSP) communication model can hinder performance increases. This is due to the complexity to handle load imbalances, to reduce serialisation imposed by blocking communication patterns, to overlap communication with computation and, finally, to deal with increasing memory overheads. The MPI specification provides advanced features such as non-blocking calls or shared memory to mitigate some of these factors. However, applying these features efficiently usually requires significant changes on the application structure.Task parallel programming models are being developed as a means of mitigating the abovementioned issues but without requiring extensive changes on the application code. In this work, we present a methodology to develop hybrid applications based on tasks called hierarchical domain over-decomposition with tasking (HDOT). This methodology overcomes most of the issues found on MPI-only and traditional hybrid MPI+OpenMP applications. However, by emphasising the reuse of data partition schemes from process-level and applying them to task-level, it enables a natural coexistence between MPI and shared-memory programming models. The proposed methodology shows promising results in terms of programmability and performance measured on a set of applications.

Analysis of World Experience in Creating Parallel Computing Systems Designed to Effectively Solve DIS-tasks

Author describes world experience in creating parallel computing systems by example Cray XE6 and network chip Gemini, designed to effectively solve Data intensive tasks (DIS-tasks). Most often, in modern supercomputers (SC), architecture options with shared (shared) memory are used to provide effective solutions to problems of high capacitive complexity, including those that contain mostly irregular work with memory. It is possible to provide support for a programming model with shared (shared) memory in various ways using hardware, as well as using virtualization software. Different options for implementing a shared memory programming model may vary in functionality and timing of memory accesses. The problem of the “<i>memory wall</i>” is that if arithmetic-logical operations take several processor cycles, then operations directly with the memory take several hundred cycles. If the memory is formed from the memories of computing nodes connected by a communication network, then the execution time of such a call includes the time of operation with the network to transfer addresses and data. This already increases the memory access time to several thousand cycles. The problem is that such delays in accessing data cause idle functional units of the processor - they cannot perform arithmetic and logical operations on data, because they simply do not exist due to the large delays in performing operations with memory.

Parallelization of Pairwise Alignment and Neighbor-Joining Algorithm in Progressive Multiple Sequence Alignment

Progressive multiple sequence alignment ClustalW is a widely used heuristic method for computing multiple sequence alignment (MSA). It has three stages: distance matrix computation using pairwise alignment, guide tree reconstruction using neighbor-joining and progressive alignment. To accelerate computing for large data, the progressive MSA algorithm needs to be parallelized. This research aims to identify, decompose and implement the pairwise alignment and neighbor-joining in progressive MSA using message passing, shared memory and hybrid programming model in the computer cluster. The experimental results obtained shared memory programming model as the best scenario implementation with speed up up to 12 times.

Automated Synthesis of Comprehensive Memory Model Litmus Test Suites

The memory consistency model is a fundamental part of any shared memory architecture or programming model. Modern weak memory models are notoriously difficult to define and to implement correctly. Most real-world programming languages, compilers, and (micro)architectures therefore rely heavily on black-box testing methodologies. The success of such techniques requires that the suite of litmus tests used to perform the testing be comprehensive--it should ideally stress all obscure corner cases of the model and of its implementation. Most litmus test suites today are generated from some combination of manual effort and randomization; however, the complex and subtle nature of contemporary memory models means that manual effort is both error-prone and subject to incomplete coverage. This paper presents a methodology for synthesizing comprehensive litmus test suites directly from a memory model specification. By construction, these suites contain all tests satisfying a minimality criterion: that no synchronization mechanism in the test can be weakened without causing new behaviors to become observable. We formalize this notion using the Alloy modeling language, and we apply it to a number of existing and newly-proposed memory models. Our results show not only that this synthesis technique can automatically reproduce all manually-generated tests from existing suites, but also that it discovers new tests that are not as well studied.

Automated Synthesis of Comprehensive Memory Model Litmus Test Suites

The memory consistency model is a fundamental part of any shared memory architecture or programming model. Modern weak memory models are notoriously difficult to define and to implement correctly. Most real-world programming languages, compilers, and (micro)architectures therefore rely heavily on black-box testing methodologies. The success of such techniques requires that the suite of litmus tests used to perform the testing be comprehensive--it should ideally stress all obscure corner cases of the model and of its implementation. Most litmus test suites today are generated from some combination of manual effort and randomization; however, the complex and subtle nature of contemporary memory models means that manual effort is both error-prone and subject to incomplete coverage. This paper presents a methodology for synthesizing comprehensive litmus test suites directly from a memory model specification. By construction, these suites contain all tests satisfying a minimality criterion: that no synchronization mechanism in the test can be weakened without causing new behaviors to become observable. We formalize this notion using the Alloy modeling language, and we apply it to a number of existing and newly-proposed memory models. Our results show not only that this synthesis technique can automatically reproduce all manually-generated tests from existing suites, but also that it discovers new tests that are not as well studied.

Software Speculation on Caching DSMs

Clusters with caching DSMs deliver programmability and performance by supporting shared-memory programming model and tolerating communication latency of remote fetches via caching. The input of a data parallel program is partitioned across machines in the cluster while the DSM transparently fetches and caches remote data as needed by the application. Irregular applications are challenging to parallelize because the input related data dependences that manifest at runtime require the use of speculation for efficiently exploiting parallelism. By speculating that there are no cross iteration dependences, multiple iterations of a data parallel loop are executed in parallel using locally cached copies of data; the absence of dependences is validated before committing the speculatively computed results. In this paper we show that in irregular data-parallel applications, while caching helps tolerate long communication latencies, using a value read from the cache in a computation can lead to misspeculation, and thus aggressive caching can degrade performance due to increased misspeculation rate. To limit misspeculation rate we present optimizations for distributed speculation on caching based DSMs that decrease the cost of misspeculation check and speed up the re-execution of misspeculated recomputations. These optimizations give speedups of 2.24\({\times }\) for graph coloring, 1.71\({\times }\) for connected components, 1.88\({\times }\) for community detection, 1.32\({\times }\) for shortest path, and 1.74\({\times }\) for pagerank over baseline parallel executions.

Hardware support for message-passing in chip multi-processors

Compared with the traditional shared-memory programming model, message passing models for chip multiprocessors (CMPs) have distinct advantages due to the relative ease of validation and the fact that they are more portable. This paper proposes a design of integrating a message-passing engine into each router of the network-on-chip as well as the programming-friendly message passing interface for these engines. Combined with the DMA mechanism, the proposed design applies the on-chip RAM as intermediary message buffer, and frees the CPU core from message-passing operations to a large extent. The detailed design and implementation, including the register-transfer-level (RTL) descriptions of the engine, are presented. Evaluations show that: compared with the software-based solution, it can decrease the message passing latency by one or two orders of magnitude. Co-simulation also demonstrates that the proposed designs effectively boost the performance of point-to-point communications on-chip, while the consumptions of power and chip-area are both limited.

Hardware support for message-passing in chip multi-processors

Compared with the traditional shared-memory programming model, message passing models for chip multiprocessors (CMPs) have distinct advantages due to the relative ease of validation and the fact that they are more portable. This paper proposes a design of integrating a message-passing engine into each router of the network-on-chip as well as the programming-friendly message passing interface for these engines. Combined with the DMA mechanism, the proposed design applies the on-chip RAM as intermediary message buffer, and frees the CPU core from message-passing operations to a large extent. The detailed design and implementation, including the register-transfer-level (RTL) descriptions of the engine, are presented. Evaluations show that: compared with the software-based solution, it can decrease the message passing latency by one or two orders of magnitude. Co-simulation also demonstrates that the proposed designs effectively boost the performance of point-to-point communications on-chip, while the consumptions of power and chip-area are both limited.

Efficient Embedded Software Migration towards Clusterized Distributed-Memory Architectures

A large portion of existing multithreaded embedded sofware has been programmed according to symmetric shared memory platforms where a monolithic memory block is shared by all cores. Such platforms accommodate popular parallel programming models such as POSIX threads and OpenMP. However with the growing number of cores in modern manycore embedded architectures, they present a bottleneck related to their centralized memory accesses. This paper proposes a solution tailored for an efficient execution of applications defined with shared-memory programming models onto on-chip distributed-memory multicore architectures. It shows how performance, area and energy consumption are significantly improved thanks to the scalability of these architectures. This is illustrated in an open-source realistic design framework, including tools from ASIC to microkernel.

A two-level parallelization method for distributed hydrological models

This paper proposes a scalable two-level parallelization method for distributed hydrological models that can use parallelizability at both the sub-basin level and the basic simulation-unit level (e.g., grid cell) simultaneously. This approach first uses the message-passing programming model to dispatch parallel tasks at the sub-basin level to different nodes with multi-core CPUs in the cluster. Each node is responsible for some of the sub-basins. Parallel tasks for each sub-basin at the basic simulation-unit level are then dispatched to multiple cores within each node using the shared-memory programming model. A grid-based distributed hydrological model was parallelized to demonstrate the performance of the proposed method, which was tested in different scenarios (e.g., different data volume, different numbers of sub-basins). Results show that the proposed two-level parallelization method had better scalability than the parallel computation at sub-basin level alone, and the parallel performance increased with data volume and the number of sub-basins.

Parallel and Scalable Short-Read Alignment on Multi-Core Clusters Using UPC++

The growth of next-generation sequencing (NGS) datasets poses a challenge to the alignment of reads to reference genomes in terms of alignment quality and execution speed. Some available aligners have been shown to obtain high quality mappings at the expense of long execution times. Finding fast yet accurate software solutions is of high importance to research, since availability and size of NGS datasets continue to increase. In this work we present an efficient parallelization approach for NGS short-read alignment on multi-core clusters. Our approach takes advantage of a distributed shared memory programming model based on the new UPC++ language. Experimental results using the CUSHAW3 aligner show that our implementation based on dynamic scheduling obtains good scalability on multi-core clusters. Through our evaluation, we are able to complete the single-end and paired-end alignments of 246 million reads of length 150 base-pairs in 11.54 and 16.64 minutes, respectively, using 32 nodes with four AMD Opteron 6272 16-core CPUs per node. In contrast, the multi-threaded original tool needs 2.77 and 5.54 hours to perform the same alignments on the 64 cores of one node. The source code of our parallel implementation is publicly available at the CUSHAW3 homepage (http://cushaw3.sourceforge.net).

Different Approaches to Parallelization of Vector Assembly

Recent developments in computer hardware bring in new opportunities in numerical mod-elling. Traditional simulation codes run sequentially on computers with a single processing unit,where only one instruction can be processed at any moment in time. The performance of single pro-cessing units is reaching the physical limits, given by transmission delays and heat build-up on thesilicon chips. The current trend in technology is parallel processing, relying on the simultaneous useof multiple processing units to solve given problem. The efficient utilization of parallel computingresources requires development of new algorithms and techniques allowing to decompose the giventask into pieces of work that can be processed simultaneously.This contribution focuses on parallelization of vector assembly operation, which is one of thecritical operations in any finite element software. The aim of presented work is to propose differentapproaches to parallelization of this operation and to evaluate their efficiency. In this contribution,we focus on shared memory programming model, where individual processes/tasks share a commonaddress space, which they read and write to asynchronously. Open Multi-Processing (OpenMP) andPortable Operating System Interface (POSIX) Threads programming models are used to implementdifferent variants of parallel assembly operations. The efficiency of implemented approaches is eval-uated on a selected benchmark problem, comparing computation times and obtained speed-ups.

Performance Analysis of an Embarrassingly Parallel Application in Atmospheric Modeling

This study aims at making a comparative study of various parallel programming models for a compute intensive application pertaining to Atmospheric modeling. Atmospheric modeling deals with predicting the behavior of atmosphere through mathematical equations governing the atmospheric fluid flows. The mathematical equations are nonlinear partial differential equations which are difficult to solve analytically. Thus fundamental governing equations of atmospheric motion are discretized into algebraic forms that are solved using numerical methods to obtain flow-field values at discrete points in time and/or space. Solving these equations often requires huge computational resource, which is normally available with high-speed supercomputers. Shallow Water equations provide a useful framework for the analysis of dynamics of large-scale atmospheric flow and for the analysis of various numerical methods that might be applied to the solution of these equations. In this study, Finite volume approach has been used for discretizing these equations that leads to a number of algebraic equations equal to the number of time instants at which the flow field values are to be evaluated. It is apparent that the application is embarrassingly parallel and its parallelization will suppress communication overhead. A High Performance Compute cluster has been employed for solving the equations involved in atmospheric modeling. Use of OpenMP and MPI APIs has paved the way to study the behavior of shared memory programming model and the message passing programming model in the context of such a highly compute intensive application. It is observed that no additional benefit can be enjoyed by creating too many software threads than the available hardware threads, as the execution resources should be shared among them.

DaSH: A benchmark suite for hybrid dataflow and shared memory programming models

Abstract The current trend in development of parallel programming models is to combine different well established models into a single programming model in order to support efficient implementation of a wide range of real world applications. The dataflow model has particularly managed to recapture the interest of the research community due to its ability to express parallelism efficiently. Thus, a number of recently proposed hybrid parallel programming models combine dataflow and traditional shared memory models. Their findings have influenced the introduction of task dependency in the OpenMP 4.0 standard. This article presents DaSH – the first comprehensive benchmark suite for hybrid dataflow and shared memory programming models. DaSH features 11 benchmarks, each representing one of the Berkeley dwarfs that capture patterns of communication and computation common to a wide range of emerging applications. DaSH also includes sequential and shared-memory implementations based on OpenMP and Intel TBB to facilitate easy comparison between hybrid dataflow implementations and traditional shared memory implementations based on work-sharing and/or tasks. Finally, we use DaSH to evaluate three different hybrid dataflow models, identify their advantages and shortcomings, and motivate further research on their characteristics.

DeNovoSync

Current shared-memory hardware is complex and inefficient. Prior work on the DeNovo coherence protocol showed that disciplined shared-memory programming models can enable more complexity-, performance-, and energy-efficient hardware than the state-of-the-art MESI protocol. DeNovo, however, severely restricted the synchronization constructs an application can support. This paper proposes DeNovoSync, a technique to support arbitrary synchronization in DeNovo. The key challenge is that DeNovo exploits race-freedom to use reader-initiated local self-invalidations (instead of conventional writer-initiated remote cache invalidations) to ensure coherence. Synchronization accesses are inherently racy and not directly amenable to self-invalidations. DeNovoSync addresses this challenge using a novel combination of registration of all synchronization reads with a judicious hardware backoff to limit unnecessary registrations. For a wide variety of synchronization constructs and applications, compared to MESI, DeNovoSync shows comparable or up to 22% lower execution time and up to 58% lower network traffic, enabling DeNovo's advantages for a much broader class of software than previously possible.