Parallel Matrix Multiplication Research Articles

WALLAX: A memristor-based Gaussian random number generator

Generating Gaussian random numbers is essential in many applications such as cryptography, games, and computer simulations. Although software Gaussian Random Number Generators (GRNG) are widely used, hardware designs have been explored for their faster speed and lower computational cost. However, hardware GRNGs usually occupy large silicon areas when implemented in Complementary Metal Oxide Semiconductor (CMOS) technology, especially for their essential Uniform Random Number Generator (URNG) part. Here, we present a memristor-based GRNG, named WALLAX, conceived from the Wallace method to generate random numbers iteratively. This GRNG circuit benefits not only from the fully parallel analog-based Vector Matrix Multiplication (VMM) feature of memristive crossbars but also harness the intrinsic stochastic switching behaviour of the memristive devices to efficiently produce truly random numbers. The vector-matrix multiplication of WALLAX is implemented on the memristive crossbar, while its random fetching step is realized by a URNG based on the stochastic switching nature of memristors. WALLAX successfully passes all the tests in the NIST 800-22 randomness test suite with 105 numbers generated and five goodness-of-fit tests with various pool sizes and effectively reduces the power and area consumption by 68.78% and 70.0% compared to digital implementations of the same GRNG method. The impact brought by memristor non-idealities is investigated by simulating the proposed structure with 1000 pools under various scenarios. Wire resistance and the stuck of state, each result in a 2.2% and 12.3% reduction in test pass rate within the tested range, respectively.

Improved parallel matrix multiplication using Strassen and Urdhvatiryagbhyam method

Improved parallel matrix multiplication using Strassen and Urdhvatiryagbhyam method

Dammann gratings-based truly parallel optical matrix multiplication accelerator.

Matrix multiplication (MM) is a fundamental operation in various scientific and engineering computations, as well as in artificial intelligence algorithms. Efficient implementation of MM is crucial for speeding up numerous applications. Photonics presents an opportunity for efficient acceleration of dense matrix computation, owing to its intrinsic advantages, such as huge parallelism, low latency, and low power consumption. However, most optical matrix computing architectures have been limited to realizing single-channel vector-matrix multiplication or using complex configurations to expand the number of channels, which does not fully exploit the parallelism of optics. In this study, we propose a novel, to the best of our knowledge, scheme for the implementation of large-scale two-dimensional optical MM with truly massive parallelism based on a specially designed Dammann grating. We demonstrate a sequence of MMs of 50 pairs of randomly generated 4 × 8 and 8 × 4 matrices in our proof-of-principle experiment. The results indicate that the mean relative error is approximately 0.048, thereby demonstrating optical robustness and high accuracy.

Development of the application for determination of the optimal partitioning form of large-dimensional matrix’s elements for multiplication on three heterogeneous processors

The article presents the results of the development of a web application for finding the optimal form of splitting matrix elements between three abstract heterogeneous processors when performing the operation of their multiplication. The paper considers five classes of parallel matrix multiplication algorithms: serial communication with a barrier, parallel communication with a barrier, serial communication with overlapping, parallel communication with overlapping, parallel overlapping with alternation. The Hockney model is used to estimate the communication complexity of the algorithms. The work uses six non-rectangular candidate partitioning shapes identified by Ashley DeFlumere in her work [1] as a result of applying the «push» technology of redistribution of matrix elements between the processors: Square Corner, Rectangle Corner, Square Rectangle, Block Rectangle, L-Rectangle, Traditional 1D Rectangular. Determination of the optimality of the form is made on the basis of mathematical models presented in the works [2,3]. The programming languages Python and JavaScript, the Django framework, the pip package manager, and Ajax technology were used to develop a web application. Solving the problem of determining the optimal matrix shape will allow for efficient planning of computing power resources for various scientific fields using parallel matrix multiplication.

Model-Based Hardware-Software Codesign of ECT Digital Processing Unit

Image reconstruction algorithm and its controller constitute the main modules of the electrical capacitance tomography (ECT) system; in order to achieve the trade-off between the attainable performance and the flexibility of the image reconstruction and control design of the ECT system, hardware-software codesign of a digital processing unit (DPU) targeting FPGA system-on-chip (SoC) is presented. Design and implementation of software and hardware components of the ECT-DPU and their integration and verification based on the model-based design (MBD) paradigm are proposed. The inner-product of large vectors constitutes the core of the majority of these ECT image reconstruction algorithms. Full parallel implementation of large vector multiplication on FPGA consumes a huge number of resources and incurs long combinational path delay. The proposed MBD of the ECT-DPU tackles this problem by crafting a parametric segmented parallel inner-product architecture so as to work as the shared hardware core unit for the parallel matrix multiplication in the image reconstruction and control of the ECT system. This allowed the parameterized core unit to be configured at system-level to tackle large matrices with the segment length working as a design degree of freedom. It allows the trade-off between performance and resource usage and determines the level of computation parallelism. Using MBD with the proposed segmented architecture, the system design can be flexibly tailored to the designer specifications to fulfill the required performance while meeting the resources constraint. In the linear-back projection image reconstruction algorithm, the segmentation scheme has exhibited high resource saving of 43% and 71% for a small degradation in a frame rate of 3% and 14%, respectively.

Efficient verification of parallel matrix multiplication in public cloud: the MapReduce case

With the advent of cloud-based parallel processing techniques, services such as MapReduce have been considered by many businesses and researchers for different applications of big data computation including matrix multiplication, which has drawn much attention in recent years. However, securing the computation result integrity in such systems is an important challenge, since public clouds can be vulnerable against the misbehavior of their owners (especially for economic purposes) and external attackers. In this paper, we propose an efficient approach using Merkle tree structure to verify the computation results of matrix multiplication in MapReduce systems while enduring an acceptable overhead, which makes it suitable in terms of scalability. Using the Merkle tree structure, we record fine-grained computation results in the tree nodes to make strong commitments for workers; they submit a commitment value to the verifier which is then used to challenge their computation results’ integrity using elected input data as verification samples. Evaluation outcomes show significant improvements comparing with the state-of-the-art technique; in case of 300*300 matrices, 73% reduction in generated proof size, 61% reduction in the proof construction time, and 95% reduction in the verification time.

HPMaX: heterogeneous parallel matrix multiplication using CPUs and GPUs

We present a novel heterogeneous parallel matrix multiplication algorithm that utilizes both central processing units (CPUs) and graphics processing units (GPUs) for large-scale matrices. Based on Strassen’s method, we represent matrix multiplication work as a set of matrix addition and multiplication tasks among their sub-matrices. Then, we distribute the tasks to CPUs and GPUs while considering the characteristics of the tasks and computing resources to minimize the data communication overhead and fully utilize the available computing power. To handle a large matrix efficiently with limited GPU memory, we also propose a block-based work decomposition method. We then further improve the performance of our method by exploiting the concurrent execution abilities of a heterogeneous parallel computing system. We implemented our method on five different heterogeneous systems and applied it to matrices of various sizes. Our method generally shows higher performance than the prior GPU-based matrix multiplication methods. Moreover, compared with the state-of-the-art GPU matrix multiplication library (i.e., CUBLAS), our method achieved up to 1.97 times higher performance using the same GPUs and CPU cores. In some cases, our method using a low-performance GPU (e.g., GTX 1060, 3 GB) achieved performance comparable to that of CUBLAS using a high-performance GPU (e.g., RTX 2080, 8 GB). Also, our method continually improves performance as we use more computing resources like additional CPU cores and GPUs. We could achieve such high performance because our approach fully utilized the capacities of the given heterogeneous parallel computing systems while employing the Strassen’s method, which has a lower asymptotic complexity. These results demonstrate the efficiency and robustness of our algorithm.

Layerwise Sparse Coding for Pruned Deep Neural Networks with Extreme Compression Ratio

Deep neural network compression is important and increasingly developed especially in resource-constrained environments, such as autonomous drones and wearable devices. Basically, we can easily and largely reduce the number of weights of a trained deep model by adopting a widely used model compression technique, e.g., pruning. In this way, two kinds of data are usually preserved for this compressed model, i.e., non-zero weights and meta-data, where meta-data is employed to help encode and decode these non-zero weights. Although we can obtain an ideally small number of non-zero weights through pruning, existing sparse matrix coding methods still need a much larger amount of meta-data (may several times larger than non-zero weights), which will be a severe bottleneck of the deploying of very deep models. To tackle this issue, we propose a layerwise sparse coding (LSC) method to maximize the compression ratio by extremely reducing the amount of meta-data. We first divide a sparse matrix into multiple small blocks and remove zero blocks, and then propose a novel signed relative index (SRI) algorithm to encode the remaining non-zero blocks (with much less meta-data). In addition, the proposed LSC performs parallel matrix multiplication without full decoding, while traditional methods cannot. Through extensive experiments, we demonstrate that LSC achieves substantial gains in pruned DNN compression (e.g., 51.03x compression ratio on ADMM-Lenet) and inference computation (i.e., time reduction and extremely less memory bandwidth), over state-of-the-art baselines.

Optimization of Clutter Simulation Based on GPU

Weibull clutter is used as an example in this paper. Based on the serial parallel analysis of Zero-memory non-linear transformation's Weibull distributed clutter algorithm, fine-grained optimization is performed. The fine-grained part uses the cuBLAS library to optimize the performance of convolution calculations. Compared with CUDA shared memory convolution method and GPU parallel matrix multiplication convolution method, its computational performance can be significantly improved under a large amount of data. Simulation results show that the Zero-memory non-linear transformation's Weibull distributed clutter simulation method is optimized and accelerated. The real-time performance of clutter data is significantly improved and its acceleration effect will be better as the amount of clutter data increases. It turns out that through fine-grained optimization, the performance of convolution calculations with large amounts of data is improved.

Cannon-type triangular matrix multiplication for the reduction of generalized HPD eigenproblems to standard form

We first develop a new variant of Cannon’s algorithm for parallel matrix multiplication on rectangular process grids. Then we tailor it to selected situations where at least one triangular matrix is involved, namely “upper triangle of (full × upper triangular),” “lower triangle of (lower triangular × upper triangular),” and “all of (upper triangular × rectangular).” These operations arise in the transformation of generalized hermitian positive definite eigenproblems AX=BXΛ to standard form A˜X˜=X˜Λ, and making use of the triangular structure enables savings in arithmetic operations and communication. Numerical results show that the new implementations outperform previously available routines, and they are particularly effective if a whole sequence of generalized eigenproblems with the same matrix B must be solved, but they can also be competitive for the solution of a single generalized eigenproblem.

FPGA Implementation of ECT Digital System for Imaging Conductive Materials

This paper presents the hardware implementation of a stand-alone Electrical Capacitance Tomography (ECT) system employing a Field Programmable Gate Array (FPGA). The image reconstruction algorithms of the ECT system demand intensive computation and fast processing of large number of measurements. The inner product of large vectors is the core of the majority of these algorithms. Therefore, a reconfigurable segmented parallel inner product architecture for the parallel matrix multiplication is proposed. In addition, hardware-software codesign targeting FPGA System-On-Chip (SoC) is applied to achieve high performance. The development of the hardware-software codesign is carried out via commercial tools to adjust the software algorithms and parameters of the system. The ECT system is used in this work to monitor the characteristic of the molten metal in the Lost Foam Casting (LFC) process. The hardware system consists of capacitive sensors, wireless nodes and FPGA module. The experimental results reveal high stability and accuracy when building the ECT system based on the FPGA architecture. The proposed system achieves high performance in terms of speed and small design density.

Square Matrix Multiplication Using CUDA on GP-GU

This paper focuses on matrix multiplication algorithm, particularly square parallel matrix multiplication using Computer Unified Device Architecture (CUDA) programming model with C programming language. Matrix multiplication is under the list of time-consuming problems that require s huge computational resources to improve its speedup. As many studies have shown, it is not easy to achieve high performance speedup in sequential matrix multiplication algorithm using larger input. The emphasis of this study is to propose a parallel algorithm to calculate the product of two square matrices with improved speedup performance compared to the sequential and OpenMP algorithms. In this research, biruni (super machine workstation) in the School of Computer Sciences, USM, Malaysia with General Purpose Graphics Processing Unit (GP-GU) was used to parallelize the matrix product algorithm. A comparison between parallel OpenMp versions and sequential algorithm with the proposed CUDA based algorithm of this research was carried out to evaluate the speedup performance of the proposed parallel CUDA based algorithm. The overall results show that CUDA based parallel matrix multiplication is approximately 400 times faster than sequential matrix multiplication and 4 times faster than OpenMp matrix multiplication algorithms, respectively. Therefore, the proposed parallel algorithm can help the researchers working with matrix multiplication application problems. It can also help mathematicians to easily calculate the product of any two matrices and obtain the result in a shorter time.

Parallel matrix multiplication circuits for use in Kalman filtering

In this work we propose several ways of the CMOS implementation of a circuit for the multiplication of matrices. We mainly focus on parallel and asynchronous solutions, however serial and mixed approaches are also discussed for the comparison. Practical applications are the motivation behind our investigations. They include fast Kalman filtering commonly used in automotive active safety functions, for example. In such filters, numerous time-consuming operations on matrices are performed. An additional problem is the growing amount of data to be processed. It results from the growing number of sensors in the vehicle as fully autonomous driving is developed. Software solutions may prove themselves to be insuffucient in the nearest future. That is why hardware coprocessors are in the area of our interests as they could take over some of the most time-consuming operations. The paper presents possible solutions, tailored to specific problems (sizes of multiplied matrices, number of bits in signals, etc.). The estimates of the performance made on the basis of selected simulation and measurement results show that multiplication of 3?3 matrices with data rate of 20 100 MSps is achievable in the CMOS 130 nm technology.

Bi-Objective Optimization of Data-Parallel Applications on Homogeneous Multicore Clusters for Performance and Energy

Performance and energy are now the most dominant objectives for optimization on modern parallel platforms composed of multicore CPU nodes. The existing intra-node and inter-node optimization methods employ a large set of decision variables but do not consider problem size as a decision variable and assume a linear relationship between performance and problem size and between energy consumption and problem size. We demonstrate using experiments of real-life data-parallel applications on modern multicore CPUs that these relationships have complex (non-linear and even non-convex) properties and, therefore, that the problem size has become an important decision variable that can no longer be ignored. This key finding motivates our work in this paper. In this paper, we first formulate the bi-objective optimization problem for performance and energy (BOPPE) for data-parallel applications on homogeneous clusters of modern multicore CPUs. It contains only one but heretofore unconsidered decision variable, the problem size. We then present an efficient and exact global optimization algorithm called <i>ALEPH</i> that solves the <i>BOPPE</i> . It takes as inputs, discrete functions of performance and dynamic energy consumption against problem size and outputs the globally Pareto-optimal set of solutions. The solutions are the workload distributions, which achieve inter-node optimization of data-parallel applications for performance and energy. While existing solvers for <i>BOPPE</i> give only one solution when the problem size and number of processors are fixed, our algorithm gives a diverse set of globally Pareto-optimal solutions. The algorithm has time complexity of <inline-formula><tex-math notation="LaTeX">$O(m^2 \times p^2)$</tex-math></inline-formula> where <inline-formula> <tex-math notation="LaTeX">$m$</tex-math></inline-formula> is the number of points in the discrete speed/energy function and <inline-formula> <tex-math notation="LaTeX">$p$</tex-math></inline-formula> is the number of available processors. We experimentally study the efficiency and scalability of our algorithm for two data parallel applications, matrix multiplication and fast Fourier transform, on a modern multicore CPU and homogeneous clusters of such CPUs. Based on our experiments, we show that the average and maximum sizes of the globally Pareto-optimal sets determined by our algorithm are 15 and 34 and 7 and 20 for the two applications respectively. Comparing with load-balanced workload distribution solution, the average and maximum percentage improvements in performance and energy respectively demonstrated for the first application are (13%,97%) and (18%,71%). For the second application, these improvements are (40%,95%) and (22%,127%). Assuming 5 percent performance degradation from the optimal is acceptable, the average and maximum improvements in energy consumption demonstrated for the two applications respectively are 9 and 44 and 8 and 20 percent. Using the algorithm and its building blocks, we also present a study of interplay between performance and energy. We demonstrate how <i>ALEPH</i> can be combined with <i> DVFS</i> -based Multi-Objective Optimization (MOP) methods to give a better set of (globally Pareto-optimal) solutions.

A New Parallel Matrix Multiplication Algorithm on Hex-Cell Network (PMMHC) Using IMAN1 Super Computer

A widespread attention has been paid in parallelizing algorithms for computationally intensive applications. In this paper, we propose a new parallel Matrix multiplication on the Hex-cell interconnection network. The proposed algorithm has been evaluated and compared with sequential algorithm in terms of speedup, and efficiency using IMAN1, where a set of simulation runs, carried out on different input data distributions with different sizes. Thus, simulation results supported the theoretical analysis and meet the expectations in which they show good performance in terms of speedup and efficiency.

Pengembangan dan Pengujian Sistem Grid Computing Menggunakan Globus Toolkit di Universitas Telkom

In this research, we build a grid computing infrastructure by utilizing existing cluster in Telkom University as back-end resources. We used middleware Globus Toolkit 6.0 and Condor 8.4.2 in developing the grid system. We tested the performance of our grid system using parallel matrix multiplication. The result showed that our grid system has achieved good performance. With the implementation of this grid system, we believe that access to high performance computing resources will become easier and the Quality of Service will also be improved.

New Model-Based Methods and Algorithms for Performance and Energy Optimization of Data Parallel Applications on Homogeneous Multicore Clusters

Modern homogeneous parallel platforms are composed of tightly integrated multicore CPUs. This tight integration has resulted in the cores contending for various shared on-chip resources such as Last Level Cache (LLC) and interconnect, leading to resource contention and non-uniform memory access (NUMA). Due to these newly introduced complexities, the performance and energy profiles of real-life scientific applications on these platforms are not smooth and may deviate significantly from the shapes that allowed traditional and state-of-the-art load balancing algorithms to minimize their computation time. In this paper, we propose new model-based methods and algorithms for minimization of time and energy of computations for the most general shapes of performance and energy profiles of data parallel applications observed on the modern homogeneous multicore clusters. We formulate the performance and energy optimization problems and present efficient algorithms of complexity O(p <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) solving these problems where p is the number of processors. It is important to note that the globally optimal solutions found by these algorithms may not load-balance the application. We experimentally study the efficiency and scalability of our algorithms for two data parallel applications, matrix multiplication and fast Fourier transform, on a modern multicore CPU and clusters of such CPUs. We also demonstrate the optimality of solutions determined by our algorithms.

A New Parallel Matrix Multiplication Algorithm on Hex-Cell Network (PMMHC) Using IMAN1 Super Computer

A widespread attention has been paid in parallelizing algorithms for computationally intensive applications. In this paper, we propose a new parallel Matrix multiplication on the Hex-cell interconnection network. The proposed algorithm has been evaluated and compared with sequential algorithm in terms of speedup, and efficiency using IMAN1, where a set of simulation runs, carried out on different input data distributions with different sizes. Thus, simulation results supported the theoretical analysis and meet the expectations in which they show good performance in terms of speedup and efficiency.

Hierarchical redesign of classic MPI reduction algorithms

Optimization of MPI collective communication operations has been an active research topic since the advent of MPI in 1990s. Many general and architecture-specific collective algorithms have been proposed and implemented in the state-of-the-art MPI implementations. Hierarchical topology-oblivious transformation of existing communication algorithms has been recently proposed as a new promising approach to optimization of MPI collective communication algorithms and MPI-based applications. This approach has been successfully applied to the most popular parallel matrix multiplication algorithm, SUMMA, and the state-of-the-art MPI broadcast algorithms, demonstrating significant multifold performance gains, especially for large-scale HPC systems. In this paper, we apply this approach to optimization of the MPI Reduce and Allreduce operations. Theoretical analysis and experimental results on a cluster of Grid’5000 platform are presented.

Algebraic methods in the congested clique

In this work, we use algebraic methods for studying distance computation and subgraph detection tasks in the congested clique model. Specifically, we adapt parallel matrix multiplication implementations to the congested clique, obtaining an $O(n^{1-2/\omega})$ round matrix multiplication algorithm, where $\omega < 2.3728639$ is the exponent of matrix multiplication. In conjunction with known techniques from centralised algorithmics, this gives significant improvements over previous best upper bounds in the congested clique model. The highlight results include: -- triangle and 4-cycle counting in $O(n^{0.158})$ rounds, improving upon the $O(n^{1/3})$ triangle detection algorithm of Dolev et al. [DISC 2012], -- a $(1 + o(1))$-approximation of all-pairs shortest paths in $O(n^{0.158})$ rounds, improving upon the $\tilde{O} (n^{1/2})$-round $(2 + o(1))$-approximation algorithm of Nanongkai [STOC 2014], and -- computing the girth in $O(n^{0.158})$ rounds, which is the first non-trivial solution in this model. In addition, we present a novel constant-round combinatorial algorithm for detecting 4-cycles.