Parallel Matrix Multiplication Research Articles

Dual-mode routing approach for photonic network on chip platforms

In this paper, a dual-mode routing approach is proposed for routing messages in 2D photonic network on chip (NoC) platforms. The dual-mode routing approach is based on selecting exclusively one of two routing techniques for each stage of a parallel application. Depending on the data exchange requirements of a specific stage of an application, either a conventional per-message-based routing or a collective routing technique is selected, and the network on chip architecture is organized to support that selection. The network on chip architecture that we use is a two-dimensional torus topology built from novel $$8\times 8$$8×8 non-blocking all-port photonic switches. The design of $$8\times 8$$8×8 non-blocking all-port photonic switch empowers each node with simultaneous connections to four other nodes in the network. The rich connectivity provided by the switches helps the collective routing technique to efficiently support collective communication operations by allowing deadlock-free and contention-free dense data exchange among the nodes. Similar dense data exchanges are also required for some commonly known parallel algorithms such as parallel matrix multiplication, linear equation solvers and n-body simulation algorithms, just to name a few. A lightweight electronic network is employed for establishing and tearing down the photonic communication paths (light paths) in per-message-based routing technique. The lightweight electronic network is also used for coordinating the whole network in the collective routing technique.

Data Partitioning on Multicore and Multi-GPU Platforms Using Functional Performance Models

Heterogeneous multiprocessor systems, which are composed of a mix of processing elements, such as commodity multicore processors, graphics processing units (GPUs), and others, have been widely used in scientific computing community. Software applications incorporate the code designed and optimized for different types of processing elements in order to exploit the computing power of such heterogeneous computing systems. In this paper, we consider the problem of optimal distribution of the workload of data-parallel scientific applications between processing elements of such heterogeneous computing systems. We present a solution that uses functional performance models (FPMs) of processing elements and FPM-based data partitioning algorithms. Efficiency of this approach is demonstrated by experiments with parallel matrix multiplication and numerical simulation of lid-driven cavity flow on hybrid servers and clusters.

Network‐aware optimization of communications for parallel matrix multiplication on hierarchical HPC platforms

SummaryCommunications on hierarchical heterogeneous high‐performance computing platforms can be optimized based on topology and performance information. For MPI, as a major programming tool for such platforms, a number of topology‐aware and performance‐aware implementations of collective operations have been proposed for optimal scheduling of messages. This approach improves performance of application and does not require to modify application source code. However, it is applicable to collective operations only and does not affect the parts of the application that are based on point‐to‐point exchanges. In this paper, we address the problem of efficient execution of data‐parallel applications on interconnected clusters and present optimizations that improve data partition by taking into account the entire communication flow of the application. This approach is also non‐intrusive to the source code but application specific. For illustration, we use parallel matrix multiplication, where the matrices are partitioned into irregular two‐dimensional rectangles assigned to different processors and arranged in columns, and the processors communicate over this partition vertically and horizontally. By rearranging the rectangles, we can minimize communications between different levels of the network hierarchy. Finding the optimal arrangement is NP‐complete; therefore, we propose two heuristic approaches based on evaluation of the communication flow on the given network topology. We demonstrate the correctness and efficiency of the proposed approaches by experimental results on multicore nodes and interconnected heterogeneous clusters. Copyright © 2015 John Wiley & Sons, Ltd.

Comparison among Performance Measures for Parallel Matrix Multiplication Algorithms

In this study we analyze how to make a proper selection for the given matrix-matrix multiplication operation and to decide which is the best suitable algorithm that generates a high throughput with a minimum time, a comparison analysis and a performance evaluation for some algorithms is carried out using the identical performance parameters.

A Parallel Matrix Multiplication Algorithm Based on Network of Moore Graph of Diameter 2

A Parallel Matrix Multiplication Algorithm Based on Network of Moore Graph of Diameter 2

Hierarchical approach to optimization of parallel matrix multiplication on large-scale platforms

Many state-of-the-art parallel algorithms, which are widely used in scientific applications executed on high-end computing systems, were designed in the twentieth century with relatively small-scale parallelism in mind. Indeed, while in 1990s a system with few hundred cores was considered a powerful supercomputer, modern top supercomputers have millions of cores. In this paper, we present a hierarchical approach to optimization of message-passing parallel algorithms for execution on large-scale distributed-memory systems. The idea is to reduce the communication cost by introducing hierarchy and hence more parallelism in the communication scheme. We apply this approach to SUMMA, the state-of-the-art parallel algorithm for matrix---matrix multiplication, and demonstrate both theoretically and experimentally that the modified Hierarchical SUMMA significantly improves the communication cost and the overall performance on large-scale platforms.

Analysis of Blocking and Scheduling for FPGA-Based Floating-Point Matrix Multiplication Analyse du blocage et de l’ordonnancement d’une multiplication matricielle à virgule flottante sur un FPGA

This paper considers blocking and scheduling for the design and implementation of field-programmable gate array (FPGA)-based floating-point parallel matrix multiplication in the presence of a memory hierarchy. For high performance, on-chip memory holds data that are reused when the computation is divided into blocks, and multiple arithmetic units perform independent operations within each block in parallel. The first contribution of this paper is a detailed analysis of the design space to characterize performance based on the amount of on-chip memory used and the approaches considered for blocking and scheduling of the computation. A comparison is also made to prior work with a unified view. The second contribution is a flexible high-performance implementation for the Altera Stratix IV EP4SGX530C2 FPGA with an interface to external double-data-rate synchronous dynamic RAM (DDR2 SDRAM) memory. Various configuration options support optimization of different objectives, and the resulting configurations have been verified in simulation and in hardware. For double-precision floating-point, a performance of 16 giga-floating-point operations per second (GFLOPS) is achievable with 64 arithmetic units at 160 MHz.

Decomposition Models of Parallel Algorithms

The article is devoted to the important role of decomposition strategy in parallel computing (parallel computers, parallel algorithms). The influence of decomposition model to performance in parallel computing we have illustrated on the chosen illustrative examples and that are parallel algorithms (PA) for numerical integration and matrix multiplication. On the basis of the done analysis of the used parallel computers in the world these are divided to the two basic groups which are from the programmer-developer point of view very different. They are also introduced the typical principal structures for both these groups of parallel computers and also their models. The paper then in an illustrative way describes the development of concrete parallel algorithm for matrix multiplication on various parallel systems. For each individual practical implementation of matrix multiplication there is introduced the derivation of its calculation complexity. The described individual ways of developing parallel matrix multiplication and their implementations are compared, analyzed and discussed from sight of programmer-developer and user in order to show the very important role of decomposition strategies mainly at the class of asynchronous parallel computers.

PERFORMANCE COMPARISON OF ROW PER SLAVE AND ROWS SET PER SLAVE METHOD IN PVM BASED PARALLEL MATRIX MULTIPLICATION

Parallel computing operates on the principle that l arge problems can often be divided into smaller one s, which are then solved concurrently to save time by taking advantage of no n-local resources and overcoming memory constraints . Multiplication of larger matrices requires a lot of computation time. This p aper deals with the two methods for handling Parall el Matrix Multiplication. First is, dividing the rows of one of the input matrices into set of rows based on the number of slaves and assigning one rows set for each slave for computation. Second method is, assigning one row of one of the input matrices at a time for each slave starting from first row to first slave and second row to second slave a nd so on and loop backs to the first slave when las t slave assignment is finished and repeated until all rows are finished assigning. The se two methods are implemented using Parallel Virtu al Machine and the computation is performed for different sizes of mat rices over the different number of nodes. The resul ts show that the row per slave method gives the optimal computation time in PVM ba sed parallel matrix multiplication.

A Formal Model For Real-Time Parallel Computation

The imposition of real-time constraints on a parallel computing environment- specifically high-performance, cluster-computing systems- introduces a variety of challenges with respect to the formal verification of the system's timing properties. In this paper, we briefly motivate the need for such a system, and we introduce an automaton-based method for performing such formal verification. We define the concept of a consistent parallel timing system: a hybrid system consisting of a set of timed automata (specifically, timed Buchi automata as well as a timed variant of standard finite automata), intended to model the timing properties of a well-behaved real-time parallel system. Finally, we give a brief case study to demonstrate the concepts in the paper: a parallel matrix multiplication kernel which operates within provable upper time bounds. We give the algorithm used, a corresponding consistent parallel timing system, and empirical results showing that the system operates under the specified timing constraints.

Fast matrix multiplication techniques based on the Adleman-Lipton model

On distributed memory electronic computers, the implementation and association of fast parallel matrix multiplication algorithms has yielded astounding results and insights. In this discourse, we use the tools of molecular biology to demonstrate the theoretical encoding of Strassen's fast matrix multiplication algorithm with DNA based on an $n$-moduli set in the residue number system, thereby demonstrating the viability of computational mathematics with DNA. As a result, a general scalable implementation of this model in the DNA computing paradigm is presented and can be generalized to the application of \emph{all} fast matrix multiplication algorithms on a DNA computer. We also discuss the practical capabilities and issues of this scalable implementation. Fast methods of matrix computations with DNA are important because they also allow for the efficient implementation of other algorithms (i.e. inversion, computing determinants, and graph theory) with DNA.

A New Parallel Matrix Multiplication Method Adapted on Fibonacci Hypercube Structure

The objective of this study was to develop a new optimal parallel algorithm for matrix multiplication which could run on a Fibonacci Hypercube structure. Most of the popular algorithms for parallel matrix multiplication can not run on Fibonacci Hypercube structure, therefore giving a method that can be run on all structures especially Fibonacci Hypercube structure is necessary for parallel matrix multiplication. For creating this method, a new model for matrix multiplication with an algorithm for data distribution on Fibonacci Hypercube structure was provided. Other than this, another optimized algorithm was designed on Mesh structure. By running the algorithms on a simulative parallel system and giving the results in graphical mode, it has been found that these two algorithms have optimized value in parallel matrix multiplication and they are more efficient than the previous algorithms.

FTPA: Supporting Fault-Tolerant Parallel Computing through Parallel Recomputing

As the size of large-scale computer systems increases, their mean-time-between-failures are becoming significantly shorter than the execution time of many current scientific applications. To complete the execution of scientific applications, they must tolerate hardware failures. Conventional rollback-recovery protocols redo the computation of the crashed process since the last checkpoint on a single processor. As a result, the recovery time of all protocols is no less than the time between the last checkpoint and the crash. In this paper, we propose a new application-level fault-tolerant approach for parallel applications called the fault-tolerant parallel algorithm (FTPA), which provides fast self-recovery. When fail-stop failures occur and are detected, all surviving processes recompute the workload of failed processes in parallel. FTPA, however, requires the user to be involved in fault tolerance. In order to ease the FTPA implementation, we developed get it fault-tolerant (GiFT), a source-to-source precompiler tool to automate the FTPA implementation. We evaluate the performance of FTPA with parallel matrix multiplication and five kernels of NAS Parallel Benchmarks on a cluster system with 1,024 CPUs. The experimental results show that the performance of FTPA is better than the performance of the traditional checkpointing approach.

Parallel algorithm for matrix multiplication based on De Bruijn network

It is necessary to develop parallel matrix multiplication because of serial matrix multiplication costing much memory and time.Many different parallel matrix multiplications have been developed according to different network topologies.De Bruijn network is one of the most attractive network models,which has good parallel computing performance.It is simple to execute parallel matrix multiplication on De Bruijn network and easy to implement.In this paper,De Bruijn network was introduced,and then a parallel algorithm for matrix multiplication based on De Bruijn network was proposed.Its speedup ratio,efficiency and scalability were analyzed.It is proved by experiments and comparison that this algorithm is equal to Cannon's in terms of time complexity.

Combining building blocks for parallel multi-level matrix multiplication

This paper presents parallel algorithms for matrix–matrix multiplication which are built up from several algorithms in a multi-level structure. The upper level consists of Strassen’s algorithm which is performed for a predefined number of recursions. The number of recursions can be adapted to the specific execution platform. The intermediate level is performed by a parallel non-hierarchical algorithm and the lower level uses efficient one-processor implementations of matrix–matrix multiplication like BLAS or ATLAS. Both the number of recursions of Strassen’s algorithm and the specific algorithms of the intermediate and lower level can be chosen so that a variety of different multi-level algorithms results. Each level of the multi-level algorithms is associated with a hierarchical partition of the set of available processors into disjoint subsets so that deeper levels of the algorithm employ smaller groups of processors in parallel. The algorithms are expressed in the multiprocessor task programming model and are coded with the runtime library Tlib. Performance experiments on several parallel platforms show that the multi-level algorithms can lead to significant performance gains.

Self-Organizing Scheduling on the Organic Grid

The Organic Grid is a biologically inspired and fully decentralized approach to the organization of computation that is based on the autonomous scheduling of strongly mobile agents on a peer-to-peer network. Through the careful design of agent behavior, the emerging organization of the computation can be customized for different classes of applications. In this paper, we report on our experience in adapting the general framework to run two representative applications on our Organic Grid prototype: the National Center for Biotechnology Information (NCBI) basic local alignment search tool (BLAST) code for sequence alignment, and the Cannon's algorithm for matrix multiplication. The first is an example of independent task application, a type of application commonly used for grid scheduling research because of its easily decomposable nature and absence of intra-node communication. The second is a popular block algorithm for parallel matrix multiplication, and represents a challenging application for grid platforms because of its highly structured and synchronous communication pattern. Agent behavior completely determines the way computation is organized on the Organic Grid. We intentionally chose two applications at opposite ends of the distributed computing spectrum having very different requirements in terms of communication topology, resource use, and response to faults. We detail the design of the agent behavior and show how the different requirements can be satisfied. By encapsulating application code and scheduling functionality into mobile agents, we decouple both computation and scheduling from the underlying grid infrastructure. In the resulting system, every node can inject a computation onto the grid; the computation naturally organizes itself around available resources.

HeteroMPI: Towards a message-passing library for heterogeneous networks of computers

The paper presents Heterogeneous MPI (HeteroMPI), an extension of MPI for programming high-performance computations on heterogeneous networks of computers. It allows the application programmer to describe the performance model of the implemented algorithm in a generic form. This model allows the specification of all the main features of the underlying parallel algorithm, which have an impact on its execution performance. These features include the total number of parallel processes, the total volume of computations to be performed by each process, the total volume of data to be transferred between each pair of the processes, and how exactly the processes interact during the execution of the algorithm. Given a description of the performance model, HeteroMPI tries to create a group of processes that executes the algorithm faster than any other group. The principal extensions to MPI are presented. We demonstrate the features of the library by performing experiments with parallel simulation of the interaction of electric and magnetic fields and parallel matrix multiplication.

Parallel matrix‐multiplication algorithm for distributed parallel computers

AbstractThis paper proposes a matrix‐multiplication algorithm suited to the distributed computer environment. The system is implemented using PVM and the execution time is evaluated. One of the conventional methods implemented using PVM is the systolic method. However, this method is limited with respect to the matrix size that can be calculated and the number of computers that can be used in the calculation. This paper first extends the algorithm so that these limitations are lifted. Then, an algorithm is proposed which can reduce the communication complexity, although the computational complexity is slightly increased. These two algorithms are implemented using PVM, and the execution time is measured. In either algorithm, an acceleration rate exceeding the number of computers is obtained. The effectiveness is found to be due to the effect of the cache memory, and the speed improvement obtained by using cache memory is investigated. The optimal number of matrix divisions from the viewpoint of cache memory size is also discussed. © 2005 Wiley Periodicals, Inc. Syst Comp Jpn, 36(4): 48–59, 2005; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.10551

A Practical Performance Comparison of Parallel Matrix Multiplication Algorithms on Network of Workstations

In this paper we practically compared the performance of three blocked based parallel multiplication algorithms with simple fixed size runtime task scheduling strategy on homogeneous cluster of workstations. Parallel Virtual Machine (PVM) was used for this study.

The problem of small and large matrices in parallel Matrix Multiplication

In this paper we discuss the case in which, using the generalized Cannon's algorithm, it is possible to reduce communications in matrix multiplication. We then apply reduction of communications to the case in which we have to multiply large matrices, in particular rectangular matrices. Two strategies are proposed to solve the problem of multiplying two large squared matrices. For the case in which we have to deal with small matrices, some methods are proposed to use the entire number of processors.