Data-parallel Programming Research Articles

Implementation and Evaluation of SIMD Instructions using RISC-V

This thesis introduces and explores the design and verification of a packed Single Instruction, Multiple Data (SIMD) instruction set for a RISC-V processor, known as the RISC-V P extension. This extension enhances the RISC-V instruction set architecture by providing packed SIMD support for 8-bit, 16-bit, and 32-bit integer data types. The significance of this architecture lies in its potential to empower developers in constructing more efficient and powerful data-parallel programs for RISC-V processors. This contribution enhances the overall capabilities of the RISC-V ecosystem, providing a valuable extension to the instruction set architecture for data-parallel applications. Keywords: Data-parallel Programs, instruction set architecture, RISC-V, P extension, Packed SIMD

When Research Topic Trend Prediction Meets Fact-Based Annotations

The unprecedented growth of publications in many research domains brings the great convenience for tracing and analyzing the evolution and development of research topics. Despite the significant contributions made by existing studies, they usually extract topics from the titles of papers, instead of obtaining topics from the authoritative sessions provided by venues (e.g., AAAI, NeurIPS, and SIGMOD). To make up for the shortcoming of existing work, we develop a novel framework namely RTTP(Research Topic Trend Prediction). Specifically, the framework contains the following two components: (1) a topic alignment strategy called TAS is designed to obtain the detailed contents of research topics in each year, (2) an enhanced prediction network called EPN is designed to capture the research trend of known years for prediction. In addition, we construct two real-world datasets of specific research domains in computer science, i.e., database and data mining, computer architecture and parallel programming. The experimental results demonstrate that the problem is well solved and our solution outperforms the state-of-the-art methods.

Heterogeneous Energy-aware Load Balancing for Industry 4.0 and IoT Environments

With the improvement of global infrastructure, Cyber-Physical Systems (CPS) have become an important component of Industry 4.0. Both the application as well as the machine work together to improve the task of interdependencies. Machine learning methods in CPS require the monitoring of computational algorithms, including adopting optimizations, fine-tuning cyber systems, improving resource utilization, as well as reducing vulnerability and also computation time. By leveraging the tremendous parallelism provided by General-Purpose Graphics Processing Units (GPGPU) as well as OpenCL, it is possible to dramatically reduce the execution time of data-parallel programs. However, when running an application with tiny amounts of data on a GPU, GPU resources are wasted because the program may not be able to fully utilize the GPU cores. This is because there is no mechanism for kernels to share a GPU due to the lack of OS support for GPUs. Optimal device selection is required to reduce the high power of the GPU. In this paper, we propose an energy reduction method for heterogeneous clustering. This study focuses on load balancing; resource-aware processor selection based on machine learning is performed using code features. The proposed method identifies energy-efficient kernel candidates (from the employment pool). Then, it selects a pair of kernel candidates from all possibilities that lead to a reduction in both energy consumption as well as execution time. Experimental results show that the proposed kernel approach reduces execution time by 2.23 times compared to a baseline scheduling system. Experiments have also shown that the execution time is 1.2 times faster than state-of-the-art approaches.

Autotuning Parallel Programs by Model Checking

The paper presents a new approach to autotuning data-parallel programs. Autotuning is a search for optimal program settings which maximize its performance. The novelty of the approach lies in the use of the model checking method to find the optimal tuning parameters by the method of counterexamples. In our work, we abstract from specific programs and specific processors by defining their representative abstract patterns. Our method of counterexamples implements the following four steps. At the first step, an execution model of an abstract program on an abstract processor is described in the language of a model checking tool. At the second step, in the language of the model checking tool, we formulate the optimality property that depends on the constructed model. At the third step, we find the optimal values of the tuning parameters by using a counterexample constructed during the verification of the optimality property. In the fourth step, we extract the information about the tuning parameters from the counter-example for the optimal parameters. We apply this approach to autotuning parallel programs written in OpenCL, a popular modern language that extends the C language for programming both standard multi-core processors (CPUs) and massively parallel graphics processing units (GPUs). As a verification tool, we use the SPIN verifier and its model representation language Promela, whose formal semantics is good for modelling the execution of parallel programs on processors with different architectures.

Code Generation from Simulink Models with Task and Data Parallelism

In this paper, we propose a method to automatically generate parallelized code from Simulink models, while exploiting both task and data parallelism. Building on previous research, we propose a model-based parallelizer (MBP) that exploits task parallelism and assigns tasks to CPU cores using a hierarchical clustering method. We also propose amethod in which data-parallel SYCL code is generated from Simulink models; computations with data parallelism are expressed in the form of S-Function Builder blocks and are executed in a heterogeneous computing environment. Most parts of the procedure can be automated with scripts, and the two methods can be applied together. In the evaluation, the data-parallel programs generated using our proposed method achieved a maximum speedup of approximately 547 times, compared to sequential programs, without observable differences in the computed results. In addition, the programs generated while exploiting both task and data parallelism were confirmed to have achieved better performance than those exploiting either one of the two.

PySchedCL: Leveraging Concurrency in Heterogeneous Data-Parallel Systems

In the past decade, high performance compute capabilities exhibited by heterogeneous GPGPU platforms have led to the popularity of data parallel programming languages such as CUDA and OpenCL. Developing high performance parallel programming solutions using such languages involve a steep learning curve due to the complexity of the underlying heterogeneous compute devices and their impact on performance. This has led to the emergence of several High Performance Computing frameworks which provide high-level abstractions for easing the development of data-parallel applications on heterogeneous platforms. However, the scheduling decisions undertaken by such frameworks only exploit coarse-grained concurrency in data parallel applications. In this paper, we propose PySchedCL, a framework which explores fine-grained concurrency aware scheduling decisions that harness the power of heterogeneous CPU/GPU architectures efficiently. We showcase the efficacy of such scheduling mechanisms over existing coarse-grained dynamic scheduling schemes by conducting extensive experimental evaluations for a diverse set of popular Deep Learning benchmarks.

SEIZE: Runtime Inspection for Parallel Dataflow Systems

Many Data-Intensive Scalable Computing (DISC) Systems provide easy-to-use functional APIs, and efficient scheduling and execution strategies allowing users to build concise data-parallel programs. In these systems, data transformations are concealed by exposed APIs, and intermediate execution states are masked under dataflow transitions. Consequently, many crucial features and optimizations (e.g., debugging, data provenance, runtime skew detection), which require runtime datafow states, are not well-supported. Inspired by our experience in implementing features and optimizations over DISC systems, we present SEIZE, a unified framework that enables dataflow inspection-wiretapping the data-path with listening logic-in MapReduce-style programming model. We generalize our lessons learned by providing a set of primitives defining dataflow inspection, orchestration options for different inspection granularities, and operator decomposition and dataflow punctuation strategy for dataflow intervention. We demonstrate the generality and flexibility of the approach by deploying SEIZE in both Apache Spark and Apache Flink, and by implementing a prototype runtime query optimizer for Spark. Our experiments show that, the overhead introduced by the inspection logic is most of the time negligible (less than 5 percent in Spark and 10 percent in Flink).

Automatic translation of data parallel programs for heterogeneous parallelism through OpenMP offloading

Heterogeneous multicores like GPGPUs are now commonplace in modern computing systems. Although heterogeneous multicores offer the potential for high performance, programmers are struggling to program such systems. This paper presents OAO, a compiler-based approach to automatically translate shared-memory OpenMP data-parallel programs to run on heterogeneous multicores through OpenMP offloading directives. Given the large user base of shared memory OpenMP programs, our approach allows programmers to continue using a single-source-based programming language that they are familiar with while benefiting from the heterogeneous performance. OAO introduces a novel runtime optimization scheme to automatically eliminate unnecessary host–device communication to minimize the communication overhead between the host and the accelerator device. We evaluate OAO by applying it to 23 benchmarks from the PolyBench and Rodinia suites on two distinct GPU platforms. Experimental results show that OAO achieves up to 32 $$\times$$ speedup over the original OpenMP version, and can reduce the host–device communication overhead by up to 99% over the hand-translated version.

Standart Dışı Değerlerin Tespitinde DBSCAN Algoritması ile Seri ve Paralel Programlama Performansının Karşılaştırılması

With the introduction of computers into our lives, digital data sizes are increasing gradually. Non-standard values (outliers) which behave differently from the others can be found in these data produced in the digital world. Detection of these values, especially in big data sets; has great importance in fields such as security, insurance, finance, medicine and genetics. Clustering methods of data mining techniques are frequently used in outlier detection in big data sets. Density based DBSCAN (Density-based spatial clustering of applications with noise) algorithm from clustering algorithms which are sensitive to noisy and outlier values is one of the most important methods in outlier detection. In this study, an application was developed using DBSCAN algorithm in C# programming language for the detection of outliers. In the developed application, 2 data sets with different data numbers were examined and analyzed. For the shortest possible data analysis time, serial and parallel programming techniques were used separately. In order to shorten the analysis time of big data sets, parallel class members in TPL (Task Parallel Library) provided with .Net 4.0 were used. In series of analysis of data sets, it was seen that DBSCAN algorithm produces more accurate results and is more practicable than other selected algorithms in terms of outlier detection. When considered in terms of computing performance, parallel programming has become more efficient as the number of data increases.

Translation of array-based loops to distributed data-parallel programs

Large volumes of data generated by scientific experiments and simulations come in the form of arrays, while programs that analyze these data are frequently expressed in terms of array operations in an imperative, loop-based language. But, as datasets grow larger, new frameworks in distributed Big Data analytics have become essential tools to large-scale scientific computing. Scientists, who are typically comfortable with numerical analysis tools but are not familiar with the intricacies of Big Data analytics, must now learn to convert their loop-based programs to distributed data-parallel programs. We present a novel framework for translating programs expressed as array-based loops to distributed data parallel programs that is more general and efficient than related work. We report on a prototype implementation on top of Spark and evaluate the performance of our system relative to hand-written programs.

Using big data computing framework and parallelized PSO algorithm to construct the reservoir dispatching rule optimization

The paper aims to study how to realize the rational allocation and efficient utilization of water resources among reservoirs and coordinate the balanced optimization of benefit among dispatching objectives under the premise of ensuring flood control safety. A multi-objective optimal dispatching system for reservoirs in Jinsha River basin based on the Spark big data computing framework and parallelized particle swarm optimization (PSO) is proposed. The characteristics of multiple objectives of water resources optimal dispatching system in Jinsha River basin are analyzed. The multiple objectives have been transformed into single objectives, and the solving model of the problem is obtained. Secondly, the parallel algorithm programming model, the PSO algorithm and its parallel strategy for solving optimization problems, and the parallel method of PSO based on Spark big data computing framework are studied. The results show that the research work provides a scientific theoretical basis and a feasible optimization method for the management and dispatch of cascade hydropower stations. Therefore, this study plays a decisive role in promoting the efficient operation of water resources optimal dispatching system and has good reference value for the development and application of big data parallel programming based on Spark platform.

A prototype framework for parallel visualization of large flow data

Scientific visualization seeks to provide deep insight into the complex pattern underlying big data, while flow visualization plays a crucial role in oceanographic-atmospheric modeling and computational fluid dynamics simulation. As an increasingly important strategy, parallel visualization incorporates data visualization with parallel computing by means of MPI (Message Passing Interface) to achieve efficient visual analysis to facilitate scientific study. This paper presents a prototype framework for parallel visualization of large flow data, involving MPI as the low-level parallel computing paradigm, DIY (Do It Yourself) as a block-oriented data-parallel programming library on top of MPI, OSUFlow as a geometry-based flow visualization engine, and VTK (Visualization Toolkit) for data input, graphics rendering, and scene interaction. It exposes the combined power of DIY and OSUFlow, including parallel yet seamless generation of streamlines as well as pathlines from vector data defined on Cartesian, rectilinear, and curvilinear grids, to a broad community of high-performance flow visualization through VTK. Preliminary results show that this framework is capable of exploiting the horsepower of a vast number of processors to accelerate data processing and visualization for explorative analysis of massive steady/unsteady volume flows.

In-Memory Data Parallel Processor

Recent developments in Non-Volatile Memories (NVMs) have opened up a new horizon for in-memory computing. Despite the significant performance gain offered by computational NVMs, previous works have relied on manual mapping of specialized kernels to the memory arrays, making it infeasible to execute more general workloads. We combat this problem by proposing a programmable in-memory processor architecture and data-parallel programming framework. The efficiency of the proposed in-memory processor comes from two sources: massive parallelism and reduction in data movement. A compact instruction set provides generalized computation capabilities for the memory array. The proposed programming framework seeks to leverage the underlying parallelism in the hardware by merging the concepts of data-flow and vector processing. To facilitate in-memory programming, we develop a compilation framework that takes a TensorFlow input and generates code for our in-memory processor. Our results demonstrate 7.5x speedup over a multi-core CPU server for a set of applications from Parsec and 763x speedup over a server-class GPU for a set of Rodinia benchmarks.

Janus: Diagnostics and reconfiguration of data parallel programs

The increasing amount of data being stored and the variety of algorithms proposed to meet processing demands of the data scientists have led to a new generation of computational environments and paradigms. These environments simplify the task of programmers, but achieving the ideal performance continues to be a challenge. In this work we investigate important factors concerning the performance of common big-data applications and consider the Spark framework as the target for our contributions. Based on that, we present the design and implementation of Janus, a tool that automates the reconfiguration of Spark applications. It leverages logs from previous executions as input, enforces configurable adjustment policies over the collected statistics and makes its decisions taking into account communication behaviors specific of the application evaluated. In order to accomplish that, Janus identifies global parameters that should be updated, or points in the user program where the data partitioning can be adjusted based on those policies. Our results show gains of up to 1.9× in the scenarios considered.

Software Speculation on Caching DSMs

Clusters with caching DSMs deliver programmability and performance by supporting shared-memory programming model and tolerating communication latency of remote fetches via caching. The input of a data parallel program is partitioned across machines in the cluster while the DSM transparently fetches and caches remote data as needed by the application. Irregular applications are challenging to parallelize because the input related data dependences that manifest at runtime require the use of speculation for efficiently exploiting parallelism. By speculating that there are no cross iteration dependences, multiple iterations of a data parallel loop are executed in parallel using locally cached copies of data; the absence of dependences is validated before committing the speculatively computed results. In this paper we show that in irregular data-parallel applications, while caching helps tolerate long communication latencies, using a value read from the cache in a computation can lead to misspeculation, and thus aggressive caching can degrade performance due to increased misspeculation rate. To limit misspeculation rate we present optimizations for distributed speculation on caching based DSMs that decrease the cost of misspeculation check and speed up the re-execution of misspeculated recomputations. These optimizations give speedups of 2.24\({\times }\) for graph coloring, 1.71\({\times }\) for connected components, 1.88\({\times }\) for community detection, 1.32\({\times }\) for shortest path, and 1.74\({\times }\) for pagerank over baseline parallel executions.

MatrixMap: Programming abstraction and implementation of matrix computation for big data analytics

The computation core of many big data applications can be expressed as general matrix computations, including linear algebra operations and irregular matrix operations. However, existing parallel programming systems such as Spark do not have programming abstraction and efficient implementation for general matrix computations. In this paper, we present MatrixMap, a unified and efficient data-parallel programming framework for general matrix computations. MatrixMap provides powerful yet simple abstraction, consisting of a distributed in-memory data structure called bulk key matrix and a programming interface defined by matrix patterns. Users can easily load data into bulk key matrices and program algorithms into parallel matrix patterns. MatrixMap outperforms current state-of-the-art systems by employing three key techniques:matrix patterns with lambda functions for irregular and linear algebra matrix operations, asynchronous computation pipeline with context-aware data shuffling strategies for specific matrix patterns and in-memory data structure reusing data in iterations. Moreover, it can automatically handle the parallelization and distribute execution of programs on a large cluster. The experiment results show that MatrixMap is 12 times faster than Spark.

Sequence Similarity Parallelization over Heterogeneous Computer Clusters Using Data Parallel Programming Model

Sequence similarity, as a special case of data intensive applications, is one of the neediest applications for parallelization. Clustered commodity computers as a cost-effective platform for distributed and parallel processing, can be leveraged to parallelize sequence similarity. However, manually designing and developing parallel programs on commodity computers is a time-consuming, complex and error-prone process. In this paper, we present a sequence similarity parallelization technique using the Apache Storm as a stream processing framework with a data parallel programming model. Storm automatically parallelizes computations via a special user-defined topology that is represented as a directed acyclic graph. The proposed technique collects streams of data from a disk and sends them sequence by sequence to clustered computers for parallel processing. We also present a dispatching policy for balancing the cluster workload and managing the cluster heterogeneity to achieve more than 99 percent parallelism. An alignment-free method, known as n-gram modeling, is used to calculate similarities between the sequences. To show the cost-performance superiority of our method on clustered commodity computers over serial processing in powerful computers, we simply use UniProtKB/SwissProt dataset for evaluation of the performance of sequence similarity as an interesting large-scale Bioinformatics application.

Designing a Tunable Nested Data-Parallel Programming System

This article describes Surge, a nested data-parallel programming system designed to simplify the porting and tuning of parallel applications to multiple target architectures. Surge decouples high-level specification of computations, expressed using a C++ programming interface, from low-level implementation details using two first-class constructs: schedules and policies. Schedules describe the valid ways in which data-parallel operators may be implemented, while policies encapsulate a set of parameters that govern platform-specific code generation. These two mechanisms are used to implement a code generation system that analyzes computations and automatically generates a search space of valid platform-specific implementations. An input and architecture-adaptive autotuning system then explores this search space to find optimized implementations. We express in Surge five real-world benchmarks from domains such as machine learning and sparse linear algebra and from the high-level specifications, Surge automatically generates CPU and GPU implementations that perform on par with or better than manually optimized versions.

A parallel pattern for iterative stencil + reduce

We advocate the Loop-of-stencil-reduce pattern as a means of simplifying the implementation of data-parallel programs on heterogeneous multi-core platforms. Loop-of-stencil-reduce is general enough to subsume map, reduce, map-reduce, stencil, stencil-reduce, and, crucially, their usage in a loop in both data-parallel and streaming applications, or a combination of both. The pattern makes it possible to deploy a single stencil computation kernel on different GPUs. We discuss the implementation of Loop-of-stencil-reduce in FastFlow, a framework for the implementation of applications based on the parallel patterns. Experiments are presented to illustrate the use of Loop-of-stencil-reduce in developing data-parallel kernels running on heterogeneous systems.

DySel

The rising pressure for simultaneously improving performance and reducing power is driving more diversity into all aspects of computing devices. An algorithm that is well-matched to the target hardware can run multiple times faster and more energy efficiently than one that is not. The problem is complicated by the fact that a program's input also affects the appropriate choice of algorithm. As a result, software developers have been faced with the challenge of determining the appropriate algorithm for each potential combination of target device and data. This paper presents DySel, a novel runtime system for automating such determination for kernel-based data parallel programming models such as OpenCL, CUDA, OpenACC, and C++AMP. These programming models cover many applications that demand high performance in mobile, cloud and high-performance computing. DySel systematically deploys candidate kernels on a small portion of the actual data to determine which achieves the best performance for the hardware-data combination. The test-deployment, referred to as micro-profiling, contributes to the final execution result and incurs less than 8% of overhead in the worst observed case when compared to an oracle. We show four major use cases where DySel provides significantly more consistent performance without tedious effort from the developer.