Automatic Parallelization Techniques Research Articles

IR-Level Dynamic Data Dependence Using Abstract Interpretation Towards Speculative Parallelization

Recently, with the wide usage of multicore architectures, automatic parallelization has become a pressing issue. Speculative parallelization, one of the most popular automatic parallelization techniques, depends on estimating probably-parallelized code parts. This in turn motivates the employment of data dependence detection techniques for these code parts to report whether they contain dependence or not in order to be parallelized. In this paper, we propose a runtime data-dependence detection technique that is based on abstract interpretation at the intermediate representation (IR) level. We apply our proposed approach on the most frequently visited blocks of the code, hot loops. Unlike most existing approaches in which data analysis occurs at compile time, our proposed method conducts the analysis immediately while interpreting the code, which in turn saves the analysis time for potentially parallelized loops. Specifically, the proposed technique depends on the concept of abstract interpretation to analyze the hot loops at runtime. This process is done by firstly computing the abstract domain for each hot loop program points. Each abstract domain is incrementally computed, till a fixpoint is achieved for all program points, and correspondingly the analysis terminates in order to consecutively detect the existence of data dependence. Once the analysis result reports a parallelization possibility for the finished hot loop, the interpreter invokes the compiler to resume the execution in a parallel fashion as recommended by our proposed approach. The proposed technique is implemented on LLVM compiler, then used to test the dependence detection for a set of kernels on the Polybench framework, and the data dependence analysis required for each kernel is studied in terms of the computation overhead.

ProCTA: program characteristic-based thread partition approach

As a thread-level automatic parallelization technique, thread-level speculation (TLS) can partition irregular serial programs into multiple threads and implement these threads in parallel on multi-core architectures to improve the performance of programs. To tackle the problem that the conventional heuristic rule-based (HR-based) thread partition approach partitions programs of different characteristics with the same scheme and several programs have bad partition results, this paper proposes a program characteristic-based thread partition approach (ProCTA), which uses a machine learning method to learn the knowledge of thread partition from TLS sample set and predicts thread partition schemes for unknown programs in accordance with programs’ characteristics and finally applies the schemes to thread partition. In Prophet compilation system, Olden benchmarks are used to evaluate ProCTA, and a comparison is made between ProCTA and conventional heuristic rules-based partition approach. The experimental results show that the proposed approach can deliver an average 18.24% speedup improvement than HR-based thread partition approach.

Qinling: A Parametric Model in Speculative Multithreading

Speculative multithreading (SpMT) is a thread-level automatic parallelization technique that can accelerate sequential programs, especially for irregular applications that are hard to be parallelized by conventional approaches. Thread partition plays a critical role in SpMT. Conventional machine learning-based thread partition approaches applied machine learning to offline guide partition, but could not explicitly explore the law between partition and performance. In this paper, we build a parametric model (Qinling) with a multiple regression method to discover the inherent law between thread partition and performance. The paper firstly extracts unpredictable parameters that determine the performance of thread partition in SpMT; secondly, we build a parametric model Qinling with extracted parameters and speedups, and train Qinling offline, as well as apply it to predict the theoretical speedups of unseen applications. Finally, validation is done. Prophet, which consists of an automatic parallelization compiler and a multi-core simulator, is used to obtain real speedups of the input programs. Olden and SPEC2000 benchmarks are used to train and validate the parametric model. Experiments show that Qinling delivers a good performance to predict speedups of unseen programs, and provides feedback guidance for Prophet to obtain the optimal partition parameters.

A hybrid sample generation approach in speculative multithreading

Speculative multithreading (SpMT) is a thread-level automatic parallelization technique to accelerate sequential programs. Machine learning has been successfully brought into SpMT to improve its performance. An appropriate sample set, which plays the role of knowledge provider, is important for machine learning-based (ML-based) thread partition. Conventionally, heuristic rules-based (HR-based) sample generation approach cannot generate adaptive samples. A hybrid sample generation approach can break this bottleneck. With this method, we firstly automatically generate samples, which are MIPS codes consisting of spawning points (SPs) and control quasi-independent points (CQIPs) by heuristic rules; secondly manually adjust the positions of SPs and CQIPs and rebuild pre-computation slice to obtain better performance for every sample; and then build model to ensure that the probability of adjusting to the optimal partition positions is increasing. During the implementation of this approach, three measures: bias weighting, preservation of optimal solutions, summary of greedy rules, are taken. In this way, we enhance the adjustment frequency for subroutines with high called time and preserve the optimal partition positions, so to achieve a stable speedup improvement. On Prophet, which is a generic SpMT processor to evaluate the performance of multithreaded programs, SPEC2000 and Olden benchmarks are used as input. Experiments show that our approach can obtain better sample sets, which deliver a better performance improvement of about 86.9% on a 16 core than the samples generated by HR-based approach. Experiment results also prove that this approach is effective to generate sample sets for ML-based thread partition.

A thread partitioning approach for speculative multithreading

Speculative multithreading (SpMT) is a thread-level automatic parallelization technique, which partitions sequential programs into multithreads to be executed in parallel. This paper presents different thread partitioning strategies for nonloops and loops. For nonloops, we propose a cost estimation based on combined run-time effects of various speculation factors to predict the resulting performance of candidate threads to guide the thread partitioning. For loops, we parallelize all the profitable loops that can potentially offer additional performance benefits by multilevel spawning in loop bodies, loop iterations, and inner loops. Then we select a proper thread boundary located in the front of loop branch instruction to reduce invalid spawning threads that waste core resources. Experimental results show that the proposed approach can obtain a significant increase in speedup and Olden benchmarks reach a performance improvement of 6.62 % on average.

PetaBricks

It is often impossible to obtain a one-size-fits-all solution for high performance algorithms when considering different choices for data distributions, parallelism, transformations, and blocking. The best solution to these choices is often tightly coupled to different architectures, problem sizes, data, and available system resources. In some cases, completely different algorithms may provide the best performance. Current compiler and programming language techniques are able to change some of these parameters, but today there is no simple way for the programmer to express or the compiler to choose different algorithms to handle different parts of the data. Existing solutions normally can handle only coarse-grained, library level selections or hand coded cutoffs between base cases and recursive cases. We present PetaBricks, a new implicitly parallel language and compiler where having multiple implementations of multiple algorithms to solve a problem is the natural way of programming. We make algorithmic choice a first class construct of the language. Choices are provided in a way that also allows our compiler to tune at a finer granularity. The PetaBricks compiler autotunes programs by making both fine-grained as well as algorithmic choices. Choices also include different automatic parallelization techniques, data distributions, algorithmic parameters, transformations, and blocking. Additionally, we introduce novel techniques to autotune algorithms for different convergence criteria. When choosing between various direct and iterative methods, the PetaBricks compiler is able to tune a program in such a way that delivers near-optimal efficiency for any desired level of accuracy. The compiler has the flexibility of utilizing different convergence criteria for the various components within a single algorithm, providing the user with accuracy choice alongside algorithmic choice.

Data-task parallelism for the VMEC program

The magnetohydrodynamic (MHD) equilibrium problem in magnetic confinement fusion devices is solved by using the variational moments equilibrium code (VMEC). The sequential version of the code is highly demanding in computational terms, requiring several days of CPU time in the most advanced computers. In this work, we present the parallelization techniques employed for this code. An automatic tool has been used to guide the data parallelization of the critical parts of the code, and some deficiencies in its automatic parallelization techniques have been determined. These shortcomings are overcome by the development of manual techniques, which offer better results. Task parallelism has also been applied to this code in an innovative way, which significantly reduces the parallel overheads due to load imbalance and message waits.

Maximizing multiprocessor performance with the SUIF compiler

This article describes automatic parallelization techniques in the SUIF (Stanford University Intermediate Format) compiler that result in good multiprocessor performance for array-based numerical programs. Parallelizing compilers for multiprocessors face many hurdles. However, SUIF's robust analysis and memory optimization techniques enabled speedups on three fourths of the NAS and SPECfp95 benchmark programs.

Supporting dynamic data structures on distributed-memory machines

Compiling for distributed-memory machines has been a very active research area in recent years. Much of this work has concentrated on programs that use arrays as their primary data structures. To date, little work has been done to address the problem of supporting programs that use pointer-based dynamic data structures. The techniques developed for supporting SPMD execution of array-based programs rely on the fact that arrays are statically defined and directly addressable. Recursive data structures do not have these properties, so new techniques must be developed. In this article, we describe an execution model for supporting programs that use pointer-based dynamic data structures. This model uses a simple mechanism for migrating a thread of control based on the layout of heap-allocated data and introduces parallelism using a technique based on futures and lazy task creation. We intend to exploit this execution model using compiler analyses and automatic parallelization techniques. We have implemented a prototype system, which we callOlden, that runs on the Intel iPSC/860 and the Thinking Machines CM-5. We discuss our implementation and report on experiments with five benchmarks.

The EM-4 under Implicit Parallelism

The EM-4 is a supercomputer that offers very fast interprocessor communication and support for multithreading. In this paper we demonstrate that the EM-4, together with an automatic parallelization technique referred to as Data-Distributed Execution (DDE), offers a computing environment in which large portions of scientific code can be executed without the need for any explicit parallelism. DDE exploits iteration-level parallelism in loops operating over arrays. It performs data-dependency analysis, based on which arrays are distributed over the different local memories. The code is then transformed to "follow" the data distribution by spawning each loop on all PEs concurrently but modifying its boundary conditions so that each operates mostly on the local subranges of the data, thus reducing remote accesses to a minimum. The approach has been tested on the EM-4 by implementing several benchmark programs representative of common scientific applications. The experiments show that high speedup is achievable by automatic parallelization of conventional Fortran-like programs.