Parallel Matrix Multiplication Algorithms Research Articles

Development of the application for determination of the optimal partitioning form of large-dimensional matrix’s elements for multiplication on three heterogeneous processors

The article presents the results of the development of a web application for finding the optimal form of splitting matrix elements between three abstract heterogeneous processors when performing the operation of their multiplication. The paper considers five classes of parallel matrix multiplication algorithms: serial communication with a barrier, parallel communication with a barrier, serial communication with overlapping, parallel communication with overlapping, parallel overlapping with alternation. The Hockney model is used to estimate the communication complexity of the algorithms. The work uses six non-rectangular candidate partitioning shapes identified by Ashley DeFlumere in her work [1] as a result of applying the «push» technology of redistribution of matrix elements between the processors: Square Corner, Rectangle Corner, Square Rectangle, Block Rectangle, L-Rectangle, Traditional 1D Rectangular. Determination of the optimality of the form is made on the basis of mathematical models presented in the works [2,3]. The programming languages Python and JavaScript, the Django framework, the pip package manager, and Ajax technology were used to develop a web application. Solving the problem of determining the optimal matrix shape will allow for efficient planning of computing power resources for various scientific fields using parallel matrix multiplication.

HPMaX: heterogeneous parallel matrix multiplication using CPUs and GPUs

We present a novel heterogeneous parallel matrix multiplication algorithm that utilizes both central processing units (CPUs) and graphics processing units (GPUs) for large-scale matrices. Based on Strassen’s method, we represent matrix multiplication work as a set of matrix addition and multiplication tasks among their sub-matrices. Then, we distribute the tasks to CPUs and GPUs while considering the characteristics of the tasks and computing resources to minimize the data communication overhead and fully utilize the available computing power. To handle a large matrix efficiently with limited GPU memory, we also propose a block-based work decomposition method. We then further improve the performance of our method by exploiting the concurrent execution abilities of a heterogeneous parallel computing system. We implemented our method on five different heterogeneous systems and applied it to matrices of various sizes. Our method generally shows higher performance than the prior GPU-based matrix multiplication methods. Moreover, compared with the state-of-the-art GPU matrix multiplication library (i.e., CUBLAS), our method achieved up to 1.97 times higher performance using the same GPUs and CPU cores. In some cases, our method using a low-performance GPU (e.g., GTX 1060, 3 GB) achieved performance comparable to that of CUBLAS using a high-performance GPU (e.g., RTX 2080, 8 GB). Also, our method continually improves performance as we use more computing resources like additional CPU cores and GPUs. We could achieve such high performance because our approach fully utilized the capacities of the given heterogeneous parallel computing systems while employing the Strassen’s method, which has a lower asymptotic complexity. These results demonstrate the efficiency and robustness of our algorithm.

Parallel Theatre: An actor framework in Java for high performance computing

Theatre is an actor-based system currently implemented in Java, which enables modelling, analysis and implementation of predictable time-dependent distributed systems like cyber-physical systems. Theatre, though, can also be used for untimed complex systems. The core feature of Theatre is its control-based character. A reflective control layer, which can be tailored to the needs of specific application domains, is responsible for scheduling and dispatching messages asynchronously exchanged among actors. A Theatre system is a federation of theatres (computing nodes), upon which application actors are partitioned. Actors can move from a theatre to another. In its distributed implementation, Theatre rests on a socket network initially established among the interacting theatres. Messages and actors exploit Java serialization/deserialization when moving from a theatre to another. This paper proposes a novel extension of Theatre, Parallel Theatre, which is developed for an exploitation of the computing potential of nowadays multi-core machines with shared memory. Parallel Theatre avoids sockets and Java serialization. A key factor of Parallel Theatre consists in being a completely lock-free computing framework. Locks are avoided not only at the application level, as it was already demonstrated by classical actor systems, but also in the runtime system, thus enabling high-performance computing of scalable systems. This paper describes Parallel Theatre and the particular control forms which were developed for untimed and timed parallel systems. Then two scalable models are presented. The first one is concerned with a parallel algorithm for matrix multiplication. The second example is related to a parallel simulation of a multi-agent model for the Iterated Prisoner's Dilemma (IPD), which permits to observe the emergence of cooperation in large populations of players. The experimental results confirm the achievement of good execution performance.

Hierarchical redesign of classic MPI reduction algorithms

Optimization of MPI collective communication operations has been an active research topic since the advent of MPI in 1990s. Many general and architecture-specific collective algorithms have been proposed and implemented in the state-of-the-art MPI implementations. Hierarchical topology-oblivious transformation of existing communication algorithms has been recently proposed as a new promising approach to optimization of MPI collective communication algorithms and MPI-based applications. This approach has been successfully applied to the most popular parallel matrix multiplication algorithm, SUMMA, and the state-of-the-art MPI broadcast algorithms, demonstrating significant multifold performance gains, especially for large-scale HPC systems. In this paper, we apply this approach to optimization of the MPI Reduce and Allreduce operations. Theoretical analysis and experimental results on a cluster of Grid’5000 platform are presented.

Comparison among Performance Measures for Parallel Matrix Multiplication Algorithms

In this study we analyze how to make a proper selection for the given matrix-matrix multiplication operation and to decide which is the best suitable algorithm that generates a high throughput with a minimum time, a comparison analysis and a performance evaluation for some algorithms is carried out using the identical performance parameters.

A Parallel Matrix Multiplication Algorithm Based on Network of Moore Graph of Diameter 2

A Parallel Matrix Multiplication Algorithm Based on Network of Moore Graph of Diameter 2

Decomposition Models of Parallel Algorithms

The article is devoted to the important role of decomposition strategy in parallel computing (parallel computers, parallel algorithms). The influence of decomposition model to performance in parallel computing we have illustrated on the chosen illustrative examples and that are parallel algorithms (PA) for numerical integration and matrix multiplication. On the basis of the done analysis of the used parallel computers in the world these are divided to the two basic groups which are from the programmer-developer point of view very different. They are also introduced the typical principal structures for both these groups of parallel computers and also their models. The paper then in an illustrative way describes the development of concrete parallel algorithm for matrix multiplication on various parallel systems. For each individual practical implementation of matrix multiplication there is introduced the derivation of its calculation complexity. The described individual ways of developing parallel matrix multiplication and their implementations are compared, analyzed and discussed from sight of programmer-developer and user in order to show the very important role of decomposition strategies mainly at the class of asynchronous parallel computers.

A Faster Parallel Algorithm for Matrix Multiplication on a Mesh Array

Matrix multiplication is a fundamental mathematical operation that has numerous applications across most scientific fields. Cannon's distributed algorithm to multiply two n-by-n matrices on a two dimensional square mesh array with n2 cells takes exactly 3n − 2 communication steps to complete. We show that it is possible to perform matrix multiplication in just 1.5n − 1 communication steps on a two dimensional square mesh array of the same size, thus halving the number of steps required.

Fast matrix multiplication techniques based on the Adleman-Lipton model

On distributed memory electronic computers, the implementation and association of fast parallel matrix multiplication algorithms has yielded astounding results and insights. In this discourse, we use the tools of molecular biology to demonstrate the theoretical encoding of Strassen's fast matrix multiplication algorithm with DNA based on an $n$-moduli set in the residue number system, thereby demonstrating the viability of computational mathematics with DNA. As a result, a general scalable implementation of this model in the DNA computing paradigm is presented and can be generalized to the application of \emph{all} fast matrix multiplication algorithms on a DNA computer. We also discuss the practical capabilities and issues of this scalable implementation. Fast methods of matrix computations with DNA are important because they also allow for the efficient implementation of other algorithms (i.e. inversion, computing determinants, and graph theory) with DNA.

A New Parallel Matrix Multiplication Method Adapted on Fibonacci Hypercube Structure

The objective of this study was to develop a new optimal parallel algorithm for matrix multiplication which could run on a Fibonacci Hypercube structure. Most of the popular algorithms for parallel matrix multiplication can not run on Fibonacci Hypercube structure, therefore giving a method that can be run on all structures especially Fibonacci Hypercube structure is necessary for parallel matrix multiplication. For creating this method, a new model for matrix multiplication with an algorithm for data distribution on Fibonacci Hypercube structure was provided. Other than this, another optimized algorithm was designed on Mesh structure. By running the algorithms on a simulative parallel system and giving the results in graphical mode, it has been found that these two algorithms have optimized value in parallel matrix multiplication and they are more efficient than the previous algorithms.

Parallel algorithm of matrix multiplication based on Biswapped network

In order to solve the problem of parallel algorithm for matrix multiplication,according to the characteristic of Biswapped network,a parallel algorithm for matrix multiplication based on Biswapped network was developed. The algorithm is simple in operation and easy to implement by using a new suggested mapping scheme. It is showed in theoretic analysis and experimental test,the proposed algorithm equals to Cannon's approximately.

Parallel algorithm for matrix multiplication based on De Bruijn network

It is necessary to develop parallel matrix multiplication because of serial matrix multiplication costing much memory and time.Many different parallel matrix multiplications have been developed according to different network topologies.De Bruijn network is one of the most attractive network models,which has good parallel computing performance.It is simple to execute parallel matrix multiplication on De Bruijn network and easy to implement.In this paper,De Bruijn network was introduced,and then a parallel algorithm for matrix multiplication based on De Bruijn network was proposed.Its speedup ratio,efficiency and scalability were analyzed.It is proved by experiments and comparison that this algorithm is equal to Cannon's in terms of time complexity.

Parallel matrix-multiplication algorithm for distributed parallel computers

This paper proposes a matrix-multiplication algorithm suited to the distributed computer environment. The system is implemented using PVM and the execution time is evaluated. One of the conventional methods implemented using PVM is the systolic method. However, this method is limited with respect to the matrix size that can be calculated and the number of computers that can be used in the calculation. This paper first extends the algorithm so that these limitations are lifted. Then, an algorithm is proposed which can reduce the communication complexity, although the computational complexity is slightly increased. These two algorithms are implemented using PVM, and the execution time is measured. In either algorithm, an acceleration rate exceeding the number of computers is obtained. The effectiveness is found to be due to the effect of the cache memory, and the speed improvement obtained by using cache memory is investigated. The optimal number of matrix divisions from the viewpoint of cache memory size is also discussed. © 2005 Wiley Periodicals, Inc. Syst Comp Jpn, 36(4): 48–59, 2005; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.10551

A Practical Performance Comparison of Parallel Matrix Multiplication Algorithms on Network of Workstations

In this paper we practically compared the performance of three blocked based parallel multiplication algorithms with simple fixed size runtime task scheduling strategy on homogeneous cluster of workstations. Parallel Virtual Machine (PVM) was used for this study.

Parallel Complexity of Matrix Multiplication

Effective design of parallel matrix multiplication algorithms relies on the consideration of many interdependent issues based on the underlying parallel machine or network upon which such algorithms will be implemented, as well as, the type of methodology utilized by an algorithm. In this paper, we determine the parallel complexity of multiplying two (not necessarily square) matrices on parallel distributed-memory machines and/or networks. In other words, we provided an achievable parallel run-time that can not be beaten by any algorithm (known or unknown) for solving this problem. In addition, any algorithm that claims to be optimal must attain this run-time. In order to obtain results that are general and useful throughout a span of machines, we base our results on the well-known LogP model. Furthermore, three important criteria must be considered in order to determine the running time of a parallel algorithm; namely, (i) local computational tasks, (ii) the initial data layout, and (iii) the communication schedule. We provide optimality results by first proving general lower bounds on parallel run-time. These lower bounds lead to significant insights on (i)–(iii) above. In particular, we present what types of data layouts and communication schedules are needed in order to obtain optimal run-times. We prove that no one data layout can achieve optimal running times for all cases. Instead, optimal layouts depend on the dimensions of each matrix, and on the number of processors. Lastly, optimal algorithms are provided.

PERFORMANCE EVALUATION OF PRACTICAL PARALLEL COMPUTER MODEL LogPQ

The present super computer will be replaced by a massively parallel computer consisting of a large number of processing elements which satisfy the continuous increasing depend for computing power. Practical parallel computing model has been expected to develop efficient parallel algorithms on massively parallel computers. Thus, we have presented a practical parallel computation model LogPQ by taking account of communication queues into the LogP model. This paper addresses the performance of a parallel matrix multiplication algorithm using LogPQ and LogP models. The parallel algorithm is implemented on Cray T3E and the parallel performances are compared with on the old machine CM-5. This shows that the communication network of T3E has superior buffering behavior than CM-5, in which we don't need to prepare extra buffering on T3E. Although, a little effect remains for both of the send and receive bufferings. On the other hand, the effect of message size remains, which shows the necessity of the overhead and gap proportional to the message size.

Hyper-systolic matrix multiplication

A novel parallel algorithm for matrix multiplication is presented. It is based on a 1-D hyper-systolic processor abstraction. The procedure can be implemented on all types of parallel systems.

To the role of decomposition strategy in high parallel algorithms

Describes the important role of decomposition strategy in parallel computing, and its influence is illustrated with the example of a very frequently used parallel algorithm for matrix multiplication. On the basis of the performed analysis of the parallel computers used worldwide divides them into the two basic groups which are considered from the programmer‐developer point of view to be very different. They also introduce the typical principal structures for both these groups of parallel computers and also their models. The paper then in an illustrative way describes the development of concrete parallel algorithm for matrix multiplication on various parallel systems. For each individual practical implementation of matrix multiplication there is introduced the derivation of its calculation complexity. The individual ways of developing parallel matrix multiplication and their implementations are compared, analysed and discussed from the point of view of the programmer‐developer and user. The very important role of the decomposition strategies that are mainly of the class of asynchronous parallel computers is highlighted.

Modeling and Simulative Performance Analysis of SMP and Clustered Computer Architectures

The performance cliaracteris tics of several classes of parallel computing systems are analyzed and com pared using high-fidelity modeling and execution- driven simulation. Processor, bus and network mod els are used to construct and simulate the architec tures of symmetric multiprocessors (SMPs), clusters of uniprocessors, and clusters of SMPs. To demon strate a typical use, the performance of ten systems is evaluated using a parallel matrix-multiplication algorithm. Because the performance of a parallel algo rithm on an architecture depends on its communica tion-to-computation ratio, an analysis of communica tion latencies for bus transactions, cache coherence, and network transactions is used to quantify each system's communication overhead. While low-level performance attributes are difficult to measure on experimental testbed systems, and are difficult to accurately represent in purely analytical models, with high fidelity simulative models they can be readily and accurately obtained. This level of detail allows the designer to rapidly prototype and evaluate the performance of parallel and distributed systems.

A flexible multicomputer algorithm for elementary matrix operations

In the present study we introduce and test a new flexible multicomputer (FM) algorithm for matrix calculations on a distributed memory computer. The FM-algorithm also performs matrix addition, subtraction, and scalar multiplication on both dense and sparse matrices. The FM-algorithm was designed to meet the need for a high-performance flexible software tool for implementing different parallel optimization algorithms. Special consideration has been taken to ensure the usability and portability of the algorithm. A preliminary flexibility test is conducted on an IBM SP2 (Cactus) machine. On the principal level, we will compare the FM-algorithm with another high-performance algorithm Summa and look at an improvement of Summa by combining it with the Strassen algorithm. On the empirical level, we will compare a chained version of the FM-algorithm with the parallel ScaLAPACK code in a set of huge matrix multiplications on a Cray T3E machine. Our results demonstrate that the FM-algorithm performs as well as the parallel ScaLAPACK code for dense matrices. FM is fully scalable for large, sparse matrices. The FM-algorithm is efficient with respect to sequential matrix multiplication. In contrast to ScaLAPACK, the fully scalable FM-algorithm is independent of mesh structure. Arbitrarily large matrices can be processed with a single processor. Scope and purpose In the paper we introduce a flexible multicomputer (FM) algorithm for elementary matrix calculations on a distributed memory computer. The algorithm is fully scalable in matrix multiplication, as shown in the study. The algorithm works as fast as the ScaLAPACK parallel matrix multiplication algorithm on a Cray T3E machine. A distinctive feature of the FM-algorithm is its independence of the number of available processors: the algorithm can process arbitrarily large matrices already on a single processor, contrary to the ScaLapack code. The reason for this possibility is that the original huge input matrices need not be represented in the CPU of the nodes. The FM-algorithm includes easy-to-use facilities for sparse matrix computations, in which blocks of zeros are eliminated from the calculation and communication tasks. The sparse matrix calculation is based on an elimination technique, where zero blocks in one input matrix and the corresponding block pairs from the other are excluded from data transmission and/or computations. The technique allows easy cell representation and facilitates information passing between nodes significantly. In comparison, the usual technique of representing each matrix cell by three numbers (the position coordinates and the cell value), leads to complicated data structures to be recognized and updated during computation. The possibility to represent specific nonzero patterns exactly, takes place at the expense of considerable communication overhead and, in most cases, little practical gain. The algorithm provides new avenues for high-performance computing, e.g., in the field of mathematical programming, where elementary matrix operations of huge dimensions are involved when solving intricate problems in engineering and economics. The FM-algorithm was designed both for architectures with shared disk memory and for architectures, where the data has to be transmitted between the processors. Naturally, in the latter system, the scalability of any matrix processing code is significantly reduced or eliminated since the communication overhead increases drastically with the problem size. However, even in this situation, the FM-algorithm needs only limited communication between nodes.