Matrix-vector Multiplication Algorithm Research Articles

Algorithms for solving a class of real quasi-symmetric Toeplitz linear systems and its applications

<abstract><p>In this paper, fast numerical methods for solving the real quasi-symmetric Toeplitz linear system are studied in two stages. First, based on an order-reduction algorithm and the factorization of Toeplitz matrix inversion, a sequence of linear systems with a constant symmetric Toeplitz matrix are solved. Second, two new fast algorithms are employed to solve the real quasi-symmetric Toeplitz linear system. Furthermore, we show a fast algorithm for quasi-symmetric Toeplitz matrix-vector multiplication. In addition, the stability analysis of the splitting symmetric Toeplitz inversion is discussed. In mathematical or engineering problems, the proposed algorithms are extraordinarily effective for solving a sequence of linear systems with a constant symmetric Toeplitz matrix. Fast matrix-vector multiplication and a quasi-symmetric Toeplitz linear solver are proven to be suitable for image encryption and decryption.</p></abstract>

Demystifying the Soft and Hardened Memory Systems of Modern FPGAs for Software Programmers through Microbenchmarking

Both modern datacenter and embedded Field Programmable Gate Arrays (FPGAs) provide great opportunities for high-performance and high-energy-efficiency computing. With the growing public availability of FPGAs from major cloud service providers such as AWS, Alibaba, and Nimbix, as well as uniform hardware accelerator development tools (such as Xilinx Vitis and Intel oneAPI) for software programmers, hardware and software developers can now easily access FPGA platforms. However, it is nontrivial to develop efficient FPGA accelerators, especially for software programmers who use high-level synthesis (HLS).The major goal of this article is to figure out how to efficiently access the memory system of modern datacenter and embedded FPGAs in HLS-based accelerator designs. This is especially important for memory-bound applications; for example, a naive accelerator design only utilizes less than 5% of the available off-chip memory bandwidth. To achieve our goal, we first identify a comprehensive set of factors that affect the memory bandwidth, including (1) the clock frequency of the accelerator design, (2) the number of concurrent memory access ports, (3) the data width of each port, (4) the maximum burst access length for each port, and (5) the size of consecutive data accesses. Then, we carefully design a set of HLS-based microbenchmarks to quantitatively evaluate the performance of the memory systems of datacenter FPGAs (Xilinx Alveo U200 and U280) and embedded FPGA (Xilinx ZCU104) when changing those affecting factors, and we provide insights into efficient memory access in HLS-based accelerator designs. Comparing between the typically used soft and hardened memory systems, respectively, found on datacenter and embedded FPGAs, we further summarize their unique features and discuss the effective approaches to leverage these systems. To demonstrate the usefulness of our insights, we also conduct two case studies to accelerate the widely used K-nearest neighbors (KNN) and sparse matrix-vector multiplication (SpMV) algorithms on datacenter FPGAs with a soft (and thus more flexible) memory system. Compared to the baseline designs, optimized designs leveraging our insights achieve about\( 3.5\times \)and\( 8.5\times \)speedups for the KNN and SpMV accelerators. Our final optimized KNN and SpMV designs on a Xilinx Alveo U200 FPGA fully utilize its off-chip memory bandwidth, and achieve about\( 5.6\times \)and\( 3.4\times \)speedups over the 24-core CPU implementations.

Space-Filling Curves for Real-Space Electronic Structure Calculations

Hamiltonian matrices for Kohn-Sham calculations implemented in real space are often large (millions by millions) but very sparse. This poses challenges and opportunities for iterative eigensolvers, which often require a large number of matrix-vector multiplications. As a consequence, an efficient parallel sparse matrix-vector multiplication algorithm is desired. Here, we investigate the benefits of using Hilbert space-filling curves (SFCs) in domain partitioning. We show that the use of Hilbert SFCs in grid-point partitioning brings better locality of the grid points, improves balance of communication, and reduces communication overhead. We also demonstrate an extension of Hilbert SFCs coupled with blockwise operations. The use of blockwise operations helps exploit the vector-processing units in contemporary computational platforms. We illustrate speedup and scalability improvements for an iterative eigensolver based on the Chebyshev-filtered subspace iteration method. Using blockwise Hilbert SFCs, we solve the Kohn-Sham problem for silicon nanocrystals up to 10 nm in diameter, which contain over 26,000 atoms. We illustrate how the density of states of silicon nanocrystals evolves to the bulk limit, where Van Hove singularities are clearly apparent.

PBBFMM3D: A parallel black-box algorithm for kernel matrix-vector multiplication

Kernel matrix-vector product is ubiquitous in many science and engineering applications. However, a naive method requires O(N2) operations, which becomes prohibitive for large-scale problems. To reduce the computation cost, we introduce a parallel method that provably requires O(N) operations and delivers an approximate result within a prescribed tolerance. The distinct feature of our method is that it requires only the ability to evaluate the kernel function, offering a black-box interface to users. Our parallel approach targets multi-core shared-memory machines and is implemented using OpenMP. Numerical results demonstrate up to 19× speedup on 32 cores. We also present a real-world application in geo-statistics, where our parallel method was used to deliver fast principle component analysis of covariance matrices.

Efficiently Solving Partial Differential Equations in a Partially Reconfigurable Specialized Hardware

Scientific computations with a wide range of applications in domains such as developing vaccines, forecasting the weather, predicting natural disasters, simulating aerodynamics of spacecraft, and exploring oil resources, create the main workloads of supercomputers. The key integration of such scientific computations is modeling physical phenomena that are done with the aid of partial differential equations (PDEs). Solving PDEs on supercomputers, even with those equipped with GPUs, consumes a large amount of power and yet is not as fast as desired. The main reason behind such slow processing is data dependency. The key challenge is that software techniques cannot resolve these dependencies, therefore, such applications cannot benefit from the parallelism provided by processors such as GPUs. Our key insight to address this challenge is that although we cannot resolve the dependencies, we can reduce their negative impacts by using hardware/software co-optimization. To this end, we propose breaking down the data-dependent operations into two groups of operations: a majority of parallelizable and the minority of data-dependent operations. We execute these two groups in the desired order: first, we put together all parallelizable operations and execute them all, subsequently; then, we switch to execute the small data-dependent part. As long as the data-dependent part is small, we can accelerate them by using fast hardware mechanisms. Besides, our proposed hardware mechanisms guarantee quickly switching between the two groups of operations. To follow the same order of execution, dictated by our software mechanism, and implemented in hardware, we also propose a new low-overhead compression format - sparsity is another attribute of PDEs that require compression. Furthermore, the core generic architecture of our proposed hardware allows the execution of other applications including sparse matrix-vector multiplication (SpMV) and graph algorithms. The key feature of the proposed hardware is partial reconfigurability, which on one hand, facilitates the execution of data-dependent computations, and on the other hand, allows executing broad application without changing the entire configuration. Our evaluations show that compared to GPUs, we achieve an average speedup of 15.6x for scientific computations while consuming 14x less energy.

P-cloth

We present a novel parallel algorithm for cloth simulation that exploits multiple GPUs for fast computation and the handling of very high resolution meshes. To accelerate implicit integration, we describe new parallel algorithms for sparse matrix-vector multiplication (SpMV) and for dynamic matrix assembly on a multi-GPU workstation. Our algorithms use a novel work queue generation scheme for a fat-tree GPU interconnect topology. Furthermore, we present a novel collision handling scheme that uses spatial hashing for discrete and continuous collision detection along with a non-linear impact zone solver. Our parallel schemes can distribute the computation and storage overhead among multiple GPUs and enable us to perform almost interactive simulation on complex cloth meshes, which can hardly be handled on a single GPU due to memory limitations. We have evaluated the performance with two multi-GPU workstations (with 4 and 8 GPUs, respectively) on cloth meshes with 0.5 -- 1.65 M triangles. Our approach can reliably handle the collisions and generate vivid wrinkles and folds at 2 -- 5 fps, which is significantly faster than prior cloth simulation systems. We observe almost linear speedups with respect to the number of GPUs.

Synthesis and Generalization of Parallel Algorithm for Matrix-vector Multiplication

Synthesis and Generalization of Parallel Algorithm for Matrix-vector Multiplication

Structured FISTA for image restoration

SummaryIn this paper, we propose an efficient numerical scheme for solving some large‐scale ill‐posed linear inverse problems arising from image restoration. In order to accelerate the computation, two different hidden structures are exploited. First, the coefficient matrix is approximated as the sum of a small number of Kronecker products. This procedure not only introduces one more level of parallelism into the computation but also enables the usage of computationally intensive matrix–matrix multiplications in the subsequent optimization procedure. We then derive the corresponding Tikhonov regularized minimization model and extend the fast iterative shrinkage‐thresholding algorithm (FISTA) to solve the resulting optimization problem. Because the matrices appearing in the Kronecker product approximation are all structured matrices (Toeplitz, Hankel, etc.), we can further exploit their fast matrix–vector multiplication algorithms at each iteration. The proposed algorithm is thus called structured FISTA (sFISTA). In particular, we show that the approximation error introduced by sFISTA is well under control and sFISTA can reach the same image restoration accuracy level as FISTA. Finally, both the theoretical complexity analysis and some numerical results are provided to demonstrate the efficiency of sFISTA.

A prewavelet-based algorithm for the solution of second-order elliptic differential equations with variable coefficients on sparse grids

We present a Ritz-Galerkin discretization on sparse grids using prewavelets, which allows us to solve elliptic differential equations with variable coefficients for dimensions d ≥ 2. The method applies multilinear finite elements. We introduce an efficient algorithm for matrix vector multiplication using a Ritz-Galerkin discretization and semi-orthogonality. This algorithm is based on standard 1-dimensional restrictions and prolongations, a simple prewavelet stencil, and the classical operator-dependent stencil for multilinear finite elements. Numerical simulation results are presented for a three-dimensional problem on a curvilinear bounded domain and for a six-dimensional problem with variable coefficients. Simulation results show a convergence of the discretization according to the approximation properties of the finite element space. The condition number of the stiffness matrix can be bounded below 10 using a standard diagonal preconditioner.

Time for dithering: fast and quantized random embeddings via the restricted isometry property

Recently, many works have focused on the characterization of non-linear dimensionality reduction methods obtained by quantizing linear embeddings, e.g., to reach fast processing time, efficient data compression procedures, novel geometry-preserving embeddings or to estimate the information/bits stored in this reduced data representation. In this work, we prove that many linear maps known to respect the restricted isometry property (RIP) can induce a quantized random embedding with controllable multiplicative and additive distortions with respect to the pairwise distances of the data points beings considered. In other words, linear matrices having fast matrix-vector multiplication algorithms (e.g., based on partial Fourier ensembles or on the adjacency matrix of unbalanced expanders) can be readily used in the definition of fast quantized embeddings with small distortions. This implication is made possible by applying right after the linear map an additive and random dither that stabilizes the impact of the uniform scalar quantization operator applied afterwards. For different categories of RIP matrices, i.e., for different linear embeddings of a metric space $(\mathcal K \subset \mathbb R^n, \ell_q)$ in $(\mathbb R^m, \ell_p)$ with $p,q \geq 1$, we derive upper bounds on the additive distortion induced by quantization, showing that it decays either when the embedding dimension $m$ increases or when the distance of a pair of embedded vectors in $\mathcal K$ decreases. Finally, we develop a novel bi-dithered quantization scheme, which allows for a reduced distortion that decreases when the embedding dimension grows and independently of the considered pair of vectors.

Reliability computing method combined with compressed storage and Krylov subspace for phased-mission system

ABSTRACTApplication of the Markov approach in the reliability analysis of phased-mission systems (PMS) may encounter the problem of state space explosion. In this paper, a relationship is derived to compute the non-zero elements of the transition rate matrix (TRM) of a Markov model on the basis of the states binary description. Moreover, a TRM compressed storage scheme (TRMCS) is proposed, and an algorithm for matrix vector multiplication under TRMCS is presented. A method to solve the state probability vector using the Krylov subspace is proposed. The performance and the effectiveness of the proposed methods are demonstrated through a typical example.

Performance analysis of distributed symmetric sparse matrix vector multiplication algorithm for multi‐core architectures

SummarySparse matrix vector multiply (SpMVM) is an important kernel that frequently arises in high performance computing applications. Due to its low arithmetic intensity, several approaches have been proposed in literature to improve its scalability and efficiency in large scale computations. In this paper, our target systems are high end multi‐core architectures and we use messaging passing interface + open multiprocessing hybrid programming model for parallelism. We analyze the performance of recently proposed implementation of the distributed symmetric SpMVM, originally developed for large sparse symmetric matrices arising in ab initio nuclear structure calculations. We study important features of this implementation and compare with previously reported implementations that do not exploit underlying symmetry. Our SpMVM implementations leverage the hybrid paradigm to efficiently overlap expensive communications with computations. Our main comparison criterion is the ‘CPU core hours’ metric, which is the main measure of resource usage on supercomputers. We analyze the effects of topology‐aware mapping heuristic using simplified network load model. We have tested the different SpMVM implementations on two large clusters with 3D Torus and Dragonfly topology. Our results show that the distributed SpMVM implementation that exploits matrix symmetry and hides communication yields the best value for the ‘CPU core hours’ metric and significantly reduces data movement overheads. Copyright © 2015 John Wiley & Sons, Ltd.

Tree-based Space Efficient Formats for Storing the Structure of Sparse Matrices

Sparse storage formats describe a way how sparse matrices are stored in a computer memory. Extensive research has been conducted about these formats in the context of performance optimization of the sparse matrix-vector multiplication algorithms, but memory efficient formats for storing sparse matrices are still under development, since the commonly used storage formats (like COO or CSR) are not sufficient. In this paper, we propose and evaluate new storage formats for sparse matrices that minimize the space complexity of information about matrix structure. The first one is based on arithmetic coding and the second one is based on binary tree format. We compare the space complexity of common storage formats and our new formats and prove that the latter are considerably more space efficient.

A Hardware-oriented Algorithm for Complex-valued Constant Matrix-vector Multiplication.

In this paper we present a hardware-oriented algorithm for constant matrix-vector product calculating, when the all elements of vector and matrix are complex numbers. The proposed algorithm versus the naive method of analogous calculations drastically reduces the number of multipliers required for FPGA implementation of complex-valued constant matrix-vector multiplication.If the fully parallel hardware implementation of naive (schoolbook) method for complex-valued matrix-vector multiplication requires 4MN multipliers, 2M N-inputs adders and 2MN two-input adders, the proposed algorithm requires only 3N(M+1)/2 multipliers and 3M(N+2)+1,5N+2 two-input adders and 3(M+1) N/2-input adders.

Finite Element Algorithms and Data Structures on Graphical Processing Units

The finite element method (FEM) is one of the most commonly used techniques for the solution of partial differential equations on unstructured meshes. This paper discusses both the assembly and the solution phases of the FEM with special attention to the balance of computation and data movement. We present a GPU assembly algorithm that scales to arbitrary degree polynomials used as basis functions, at the expense of redundant computations. We show how the storage of the stiffness matrix affects the performance of both the assembly and the solution. We investigate two approaches: global assembly into the CSR and ELLPACK matrix formats and matrix-free algorithms, and show the trade-off between the amount of indexing data and stiffness data. We discuss the performance of different approaches in light of the implicit caches on Fermi GPUs and show a speedup over a two-socket 12-core CPU of up to 10 times in the assembly and up to 6 times in the solution phase. We present our sparse matrix-vector multiplication algorithms that are part of a conjugate gradient iteration and show that a matrix-free approach may be up to two times faster than global assembly approaches and up to 4 times faster than NVIDIA's cuSPARSE library, depending on the preconditioner used.

Efficient edit distance with duplications and contractions

We propose three algorithms for string edit distance with duplications and contractions. These include an efficient general algorithm and two improvements which apply under certain constraints on the cost function. The new algorithms solve a more general problem variant and obtain better time complexities with respect to previous algorithms. Our general algorithm is based on min-plus multiplication of square matrices and has time and space complexities of O (|Σ|MP (n)) and O (|Σ|n2), respectively, where |Σ| is the alphabet size, n is the length of the strings, and MP (n) is the time bound for the computation of min-plus matrix multiplication of two n × n matrices (currently, due to an algorithm by Chan).For integer cost functions, the running time is further improved to . In addition, this variant of the algorithm is online, in the sense that the input strings may be given letter by letter, and its time complexity bounds the processing time of the first n given letters. This acceleration is based on our efficient matrix-vector min-plus multiplication algorithm, intended for matrices and vectors for which differences between adjacent entries are from a finite integer interval D. Choosing a constant , the algorithm preprocesses an n × n matrix in time and space. Then, it may multiply the matrix with any given n-length vector in time. Under some discreteness assumptions, this matrix-vector min-plus multiplication algorithm applies to several problems from the domains of context-free grammar parsing and RNA folding and, in particular, implies the asymptotically fastest time algorithm for single-strand RNA folding with discrete cost functions.Finally, assuming a different constraint on the cost function, we present another version of the algorithm that exploits the run-length encoding of the strings and runs in time and space, where is the length of the run-length encoding of the strings.

A Methodology for Speeding up MVM for Regular, Toeplitz and Bisymmetric Toeplitz Matrices

The Matrix Vector Multiplication algorithm is an important kernel in most varied domains and application areas and the performance of its implementations highly depends on the memory utilization and data locality. In this paper, a new methodology for MVM including different types of matrices, i.e. Regular, Toeplitz and Bisymmetric Toeplitz, is presented in detail. This methodology achieves higher execution speed than the software state of the art library, ATLAS (speedup from 1.2 up to 4.4), and other conventional software implementations, for both general (SIMD unit is used) and embedded processors. This is achieved by fully and simultaneously exploiting the combination of software and hardware parameters as one problem and not separately.

Wavefront reconstruction for extremely large telescopes via CuRe with domain decomposition

The Cumulative Reconstructor is an accurate, extremely fast reconstruction algorithm for Shack-Hartmann wavefront sensor data. But it has shown an unacceptable high noise propagation for large apertures. Therefore, in this paper we describe a domain decomposition approach to deal with this drawback. We show that this adaptation of the algorithm gives the same reconstruction quality as the original algorithm and leads to a significant improvement with respect to noise propagation. The method is combined with an integral control and compared to the classical matrix vector multiplication algorithm on an end-to-end simulation of a single conjugate adaptive optics system. The reconstruction time is 20n (number of subapertures), and the method is parallelizable.

Comparing Different Sparse Matrix Storage Structures as Index Structure for Arabic Text Collection

In the authors’ study they evaluate and compare the storage efficiency of different sparse matrix storage structures as index structure for Arabic text collection and their corresponding sparse matrix-vector multiplication algorithms to perform query processing in any Information Retrieval (IR) system. The study covers six sparse matrix storage structures including the Coordinate Storage (COO), Compressed Sparse Row (CSR), Compressed Sparse Column (CSC), Block Coordinate (BCO), Block Sparse Row (BSR), and Block Sparse Column (BSC). Evaluation depends on the storage space requirements for each storage structure and the efficiency of the query processing algorithm. The experimental results demonstrate that CSR is more efficient in terms of storage space requirements and query processing time than the other sparse matrix storage structures. The results also show that CSR requires the least amount of disk space and performs the best in terms of query processing time compared with the other point entry storage structures (COO, CSC). The results demonstrate that BSR requires the least amount of disk space and performs the best in terms of query processing time compared with the other block entry storage structures (BCO, BSC).

Performance Analysis of MatrixVector Multiplication in Hybrid 'MPI + OpenMP'

g of multiple tasks simultaneously on multiple processors is called Parallel Computing. The parallel program consists of multiple active processes simultaneously solving a given problem. Parallel computers can be roughly classified as Multi-Processor and Multi-Core. In both these classifications the hardware supports parallelism with computer node having multiple processing elements in a single machine, either in single chip pack or on more than one distinct chip respectively. Parallel programming is the ability of program to run on this infrastructure which is still quite difficult and complex task to achieve. Out of many two different approaches used in parallel environment are MPI and OpenMP, each one of them having their own merits and demerits. Hybrid model combines both approaches in the pursuit of reducing the weaknesses in individual. In proposed approach takes a pair of, Matrices produces another matrix by using Matrix-Vector Multiplication Algorithm. The resulting matrix agrees with the result of composition of the linear transformations represented by the two original matrices. This algorithm is implemented in MPI, OpenMP, and Hybrid mode. The algorithm is tested for number of nodes with different number of matrix size. The results indicates that the Hybrid approach out performs the MPI and OpenMP approach.