Irregular Codes Research Articles

Runtime characterisation of irregular accesses applied to parallelisation of irregular reductions

Irregular reduction operations are the core of many large scientific and engineering applications. There are, in the literature, different methods to solve these operations in parallel. In this paper we discuss a new technique which improves performance significantly, both in terms of execution time and memory overhead. These improvements are achieved in the preprocessing as well as in the resulting parallel code. Our proposal is based on the use of the Irregular Access Region Descriptor (IARD). This data structure is a compact characterisation of indirectly accessed arrays that can be used for the efficient parallelisation of a wide spectrum of irregular codes. In this paper we present its application to parallelise irregular reduction operations on a shared memory machine. We compare our proposal with two other competitive solutions.

A More Accurate One-Dimensional Analysis and Design of Irregular LDPC Codes

We introduce a new one-dimensional (1-D) analysis of low-density parity-check (LDPC) codes on additive white Gaussian noise channels which is significantly more accurate than similar 1-D methods. Our method assumes a Gaussian distribution in message-passing decoding only for messages from variable nodes to check nodes. Compared to existing work, which makes a Gaussian assumption both for messages from check nodes and from variable nodes, our method offers a significantly more accurate estimate of convergence behavior and threshold of convergence. Similar to previous work, the problem of designing irregular LDPC codes reduces to a linear programming problem. However, our method allows irregular code design in a wider range of rates without any limit on the maximum variable-node degree. We use our method to design irregular LDPC codes with rates greater than 1/4 that perform within a few hundredths of a decibel from the Shannon limit. The designed codes perform almost as well as codes designed by density evolution.

Improved Progressive-Edge-Growth (PEG) Construction of Irregular LDPC Codes

Progressive-edge-growth (PEG) algorithm is known to construct low-density parity-check codes at finite block lengths with very good performance. In this letter, we propose a very simple modification to PEG construction for irregular codes, which considerably improves the performance at high signal-to-noise (SNR) ratios with no sacrifice in low-SNR performance.

Threshold of LDPC-Coded BICM for Rayleigh Fading

We analyze the threshold and convergence of LDPC coded bit interleaved coded modulation over Rayleigh fading channels. We compute the system threshold using the discretized density evolution and the extrinsic information transfer (EXIT) chart, and verify its effectiveness by simulation. Simulated detector output pdf is used to compute the variable node EXIT curve. It is shown that the convergence SNR observed from the transfer curves matches the density evolution threshold perfectly, and the EXIT chart actually provides a more computational efficient approach to estimate the threshold. An irregular code design example using the EXIT chart is given.

Asymptotic Enumeration Methods for Analyzing LDPC Codes

We show how asymptotic estimates of powers of polynomials with nonnegative coefficients can be used in the analysis of low-density parity-check (LDPC) codes. In particular, we show how these estimates can be used to derive the asymptotic distance spectrum of both regular and irregular LDPC code ensembles. We then consider the binary erasure channel (BEC). Using these estimates we derive lower bounds on the error exponent, under iterative decoding, of LDPC codes used over the BEC. Both regular and irregular code structures are considered. These bounds are compared to the corresponding bounds when optimal (maximum-likelihood (ML)) decoding is applied.

Design of Serially Concatenated Systems Depending on the Block Length

Based on extrinsic information transfer (EXIT) charts, the convergence behavior of iterative decoding is studied for a number of serially concatenated systems, such as a serially concatenated code, coded data transmission over an intersymbol interference channel, bit-interleaved coded modulation, or trellis-coded modulation. Efficient optimization algorithms based on simplified EXIT chart construction are devised to find irregular codes improving the convergence of iterative decoding. One optimization criterion is to find concatenated systems exhibiting thresholds of successful decoding convergence, which are close to information-theoretic limits. However, these thresholds are approached only for very long block lengths. To overcome this problem, the decoding convergence after a fixed, finite number of iterations is optimized, which yields systems performing very well for short block lengths, too. As an example, optimal system configurations for communication over an additive white Gaussian noise channel are presented.

Irregular low-density parity-check codes: construction and performance on perpendicular magnetic recording channels

In this paper, we propose an algorithm to design an irregular low-density parity-check code (LDPC), with a given degree distribution pair, from a random regular LDPC code. The proposed algorithm splits columns and rows of the regular code in order to achieve the desired degree distribution pair. Kou, Lin, and Fossorier have used column splitting to increase code rate. However, we use the splitting technique to achieve a given degree distribution pair in the bipartite graph of the resultant irregular code. The bit-error rate (BER) performance of high-rate irregular LDPC codes, constructed using this algorithm, were simulated on equalized perpendicular magnetic recording channels. In addition, this paper compares the performance of these irregular codes with that of random regular LDPC codes.

Optimization techniques for parallel irregular reductions

Different parallelization techniques have been proposed in the literature for irregular reductions in the context of shared memory multiprocessors. They may be classified into two broad families: those based on privatization of the reduction arrays and those based on the partitioning of the reduction arrays. Methods in the first family are simple but no data locality is exploited and their memory scalability is low. On the other hand, methods in the second family are more complex as they require an inspection phase but they exploit data locality and scale up better in memory. Focusing on partitioning-based methods, although they exhibit a good performance in a wide variety of irregular codes, some specific input data patterns may exist for which the performance is lowered. In particular these kind of access patterns may reduce the exploited parallelism by the method or introduce workload unbalances. In order to mitigate these negative effects, we propose three optimizations for a specific partitioning-based method (DWA–LIP). These optimizations try to increase the exploited parallelism, balance the workload and reduce the effect of high contention degree regions in the reduction arrays. Efficient implementations of the proposed optimizations for the DWA–LIP method have been tested experimentally, and compared with other methods for parallelizing irregular reductions.

Modified belief-propagation algorithm for decoding of irregular low-density parity-check codes

A modified belief-propagation algorithm is proposed for decoding of irregular low-density parity-check codes. This algorithm effectively reduces the decoding complexity while achieving a BER performance near to or even better than that of the standard belief-propagation algorithm.

Tailbiting codes: bounds and search results

Tailbiting trellis representations of linear block codes with an arbitrary sectionalization of the time axis are studied. The notations of regular and irregular tailbiting codes are introduced and their maximal state complexities are lower-bounded. The asymptotic behavior of the derived bound is investigated. Furthermore, for regular tailbiting codes the product state complexity is lower-bounded. Tables of new tailbiting trellis representations of linear block codes of rates 1/2, 1/3, and 1/4 are presented. Almost all found trellises are optimal in the sense of the new bound on the state complexity and for most codes with nonoptimal trellises there exist time-varying trellises which are optimal. Five of our newly found tailbiting codes are better than the previously known linear codes with the same parameters. Four of them are also superior to any previously known nonlinear code with the same parameters. Also, more than 40 other quasi-cyclic codes have been found that improve the parameter set of previously known quasi-cyclic codes.

Exploiting memory affinity in OpenMP through schedule reuse

In this paper we explore the idea of reusing loop schedules to improve the scalability of numerical codes in shared-memory architectures with non-uniform memory access. The main objective is to implicitly construct affinity links between threads and data accesses and reuse them as much as possible along the execution of the program. These links are created through the definition and reuse of iteration schedules which are either defined statically by the user or created dynamically at run time. The paper does not include a formal proposal of OpenMP extensions but includes some experiments showing the usefulness of constructing affinity links in both regular and irregular codes.

Improved low-density parity-check codes using irregular graphs

We construct new families of error-correcting codes based on Gallager's (1973) low-density parity-check codes. We improve on Gallager's results by introducing irregular parity-check matrices and a new rigorous analysis of hard-decision decoding of these codes. We also provide efficient methods for finding good irregular structures for such decoding algorithms. Our rigorous analysis based on martingales, our methodology for constructing good irregular codes, and the demonstration that irregular structure improves performance constitute key points of our contribution. We also consider irregular codes under belief propagation. We report the results of experiments testing the efficacy of irregular codes on both binary-symmetric and Gaussian channels. For example, using belief propagation, for rate 1/4 codes on 16000 bits over a binary-symmetric channel, previous low-density parity-check codes can correct up to approximately 16% errors, while our codes correct over 17%. In some cases our results come very close to reported results for turbo codes, suggesting that variations of irregular low density parity-check codes may be able to match or beat turbo code performance.

Efficient compiler and run-time support for parallel irregular reductions

Many scientific applications are comprised of irregular reductions on large data sets. In shared-memory parallel programs, these irregular reductions are typically computed in parallel using replicated buffers, then combined using synchronization. We develop L ocalW rite, a new technique which partitions irregular reductions so that each processor computes values only for locally assigned data, eliminating the need for buffers or synchronized writes. Computation is replicated if its results are needed on multiple processors. We experimentally evaluate its performance for three irregular codes on a software DSM running on a distributed-memory multiprocessor and two shared-memory multiprocessors while varying connectivity, locality, and adaptivity. Results show L ocalW rite improves performance significantly compared to using replicated buffers, and can match or exceed explicit message-passing gather/scatter for applications with low locality or high adaptivity.

Statistical physics of irregular low-density parity-check codes

Low-density parity-check codes with irregular constructions have been recently shown to outperform the most advanced error-correcting codes to date. In this paper we apply methods of statistical physics to study the typical properties of simple irregular codes. We use the replica method to find a phase transition which coincides with Shannon's coding bound when appropriate parameters are chosen. The decoding by belief propagation is also studied using statistical physics arguments; the theoretical solutions obtained are in good agreement with simulations. We compare the performance of irregular with that of regular codes and discuss the factors that contribute to the improvement in performance.

Limits of task-based parallelism in irregular applications

Tomorrow's microprocessors will be able to handle multiple flows of control. Applications that exhibit task level parallelism (TLP) and can be decomposed into parallel tasks will perform well on these platforms. TLP arises when a task is independent of its neighboring code. Traditional parallel compilers exploit one variety of TLP, loop level parallelism (LLP), where loop iterations are executed in parallel. LLP can overwhelming be found in numeric, typically FORTRAN programs with regular patterns of data accesses. In contrast, irregular applications, typified by general purpose integer applications, exhibit little LLP as they tend to access data in irregular patterns through pointers. Without pointer disambiguation to analyze data access dependences, traditional parallel compilers cannot parallelize these irregular applications and ensure correct execution.We focus on a different variety of TLP, namely Speculative Task Parallelism (STP). STP arises when a task (either a leaf-procedure, a non-leaf procedure or an entire loop) is control- and memory-independent of its preceding code, and thus could be executed in parallel. Two sections of code are memory-independent when neither contains a store to a memory location that the other accesses. To exploit STP, we assume a hypothetical speculative machine that supports speculative futures (a parallel programming construct that executes a task early on a different thread or processor) with mechanisms for resolving incorrect speculation when the task is not, after all, independent. This allows us to speculatively parallelize code when there is a high probability of independence, but no guarantee.Figure 1 illustrates STP, showing a task Y in the dynamic instruction stream of an irregular application that has no memory access conflicts with a group of instructions, X, that precede Y. The shorter of X and Y determines the overlap of memory-independent instructions as seen in Figures 1(b) and 1(c). In the absence of any register dependences, X and Y may be executed in parallel, resulting in shorter execution time. It is hard for traditional parallel compilers of pointer-based languages to expose this parallelism.The goals of this paper are to identify such regions as X and Y within irregular applications and to find the number of instructions that may thus be removed from the critical path. This number represents the maximum STP when the cost of exploiting STP is zero.Because the biggest barrier to detecting independence in irregular codes is memory disambiguation, we identify memory-independent tasks using a profile-based approach and measure the amount of STP by estimating the amount of memory-independent instructions those tasks expose. We vary the level of control dependence and memory dependence to investigate their effect on the amount of memory-independence we find. We profile at different memory granularities and introduce synchronization to expose higher levels of memory-independence. Across this variety of speculation assumptions, 7 to 22% of dynamic instructions are within tasks that are found to be memory-independent. This was on the SPECint95 benchmarks, a set of irregular applications for which traditional methods of parallelization are ineffective.

Porting and performance evaluation of irregular codes using OpenMP

In the last two years, OpenMP has been gaining popularity as a standard for developing portable shared memory parallel programs. With the improvements in centralized shared memory technologies and the emergence of distributed shared memory (DSM) architectures, several medium-to-large physical and logical shared memory configurations are now available. Thus, OpenMP stands to be a promising medium for developing scalable and portable parallel programs. In this paper, we focus on evaluating the suitability of OpenMP for developing scalable and portable irregular applications. We examine the programming paradigms supported by OpenMP that are suitable for this important class of applications, the performance and scalability achieved with these applications, the achieved locality and uniprocessor cache performance and the factors behind imperfect scalability. We have used two irregular applications and one NAS irregular code as the benchmarks for our study. Our experiments have been conducted on a 64-processor SGI Origin 2000. Our experiments show that reasonably good scalability is possible using OpenMP if careful attention is paid to locality and load balancing issues. Particularly, using the Single Program Multiple Data (SPMD) paradigm for programming is a significant win over just using loop parallelization directives. As expected, the cost of remote accesses is the major factor behind imperfect speedups of SPMD OpenMP programs. Copyright © 2000 John Wiley & Sons, Ltd.

Data-parallel support for numerical irregular problems

A large class of intensive numerical applications show an irregular structure, exhibiting an unpredictable runtime behavior. Two kinds of irregularity can be distinguished in these applications. First, irregular control structures, derived from the use of conditional statements on data only known at runtime. Second, irregular data structures, derived from computations involving sparse matrices, grids, trees, graphs, etc. Many of these applications exhibit a large amount of parallelism, but the above features usually make that exploiting such parallelism becomes a very difficult task. This paper discusses the effective parallelization of numerical irregular codes, focusing on the definition and use of data-parallel extensions to express the parallelism that they exhibit. We show that the combination of data distributions with storage structures allows to obtain efficient parallel codes. Codes dealing with sparse matrices, finite element methods and molecular dynamics (MD) simulations are taken as working examples.

DATA DISTRIBUTION ANALYSIS FOR IRREGULAR AND ADAPTIVE CODES

An important component in compiling for distributed memory machines is data partitioning. While a number of automatic analysis techniques have been proposed for this phase, none of them is applicable for irregular problems. In this paper, we present compile-time analysis for determining data partitioning for such applications. We have developed a set of cost functions for determining communication and redistribution costs in irregular codes. We first determine the appropriate distributions for a single data parallel statement, and then use the cost functions with a greedy algorithm for computing distributions for the full program. Initial performance results on a 16 processor IBM SP-2 are also presented.

VFC: The Vienna Fortran Compiler

High Performance Fortran (HPF) offers an attractive high‐level language interface for programming scalable parallel architectures providing the user with directives for the specification of data distribution and delegating to the compiler the task of generating an explicitly parallel program. Available HPF compilers can handle regular codes quite efficiently, but dramatic performance losses may be encountered for applications which are based on highly irregular, dynamically changing data structures and access patterns. In this paper we introduce the Vienna Fortran Compiler (VFC), a new source‐to‐source parallelization system for HPF+, an optimized version of HPF, which addresses the requirements of irregular applications. In addition to extended data distribution and work distribution mechanisms, HPF+ provides the user with language features for specifying certain information that decisively influence a program’s performance. This comprises data locality assertions, non‐local access specifications and the possibility of reusing runtime‐generated communication schedules of irregular loops. Performance measurements of kernels from advanced applications demonstrate that with a high‐level data parallel language such as HPF+ a performance close to hand‐written message‐passing programs can be achieved even for highly irregular codes.

Modeling set associative caches behavior for irregular computations

While much work has been devoted to the study of cache behavior during the execution of codes with regular access patterns, little attention has been paid to irregular codes. An important portion of these codes are scientific applications that handle compressed sparse matrices. In this work a probabilistic model for the prediction of the number of misses on a K -way associative cache memory considering sparse matrices with a uniform or banded distribution is presented. Two different irregular kernels are considered: the sparse matrix-vector product and the transposition of a sparse matrix. The model was validated with simulations on synthetic uniform matrices and banded matrices from the Harwell-Boeing collection.