Matrix Multiplication Algorithm Research Articles

Investigation of Energy and Power Characteristics of Various Matrix Multiplication Algorithms

Investigation of Energy and Power Characteristics of Various Matrix Multiplication Algorithms

Research on fast matrix multiplication algorithm

This research mainly focuses on fast matrix multiplication algorithms. Fast matrix multiplication is one of the most fundamental problems in computer science. The fast matrix multiplication algorithm differs from conventional matrix multiplication in that it offers a faster computational approach that can perform the operation in less than O(n3) time complexity. This algorithm provides a more efficient method for multiplying matrices, significantly reducing the computational requirements. The Laser method, developed by Coppersmith and Winograd, is an algorithm for matrix multiplication that does not involve direct computation. It establishes a relationship between matrix multiplication and tensors and simplifies the operation by finding an intermediate tensor that is computationally manageable. This method applies a series of simplification operations to determine an upper bound on the computational complexity of matrix multiplication. However, as matrices become larger, the computational and memory requirements increase, posing challenges for practical implementation. This research will present the main ideas and performance of the Laser method and discuss the improvements made to the Laser method, including refined analysis and asymmetric hashing techniques. Additionally, it highlights the need for further exploration, such as parallel computing and optimization strategies, to enhance the efficiency of matrix multiplication algorithms. Furthermore, this research will also provide a prospectus for the future of matrix multiplication algorithms, such as the practical implementation of the Laser method.

Can Neural Networks Do Arithmetic? A Survey on the Elementary Numerical Skills of State-of-the-Art Deep Learning Models

Creating learning models that can exhibit sophisticated reasoning abilities is one of the greatest challenges in deep learning research, and mathematics is rapidly becoming one of the target domains for assessing scientific progress in this direction. In the past few years there has been an explosion of neural network architectures, datasets, and benchmarks specifically designed to tackle mathematical problems, reporting impressive achievements in disparate fields such as automated theorem proving, numerical integration, and the discovery of new conjectures or matrix multiplication algorithms. However, despite this notable success it is still unclear whether deep learning models possess an elementary understanding of quantities and numbers. This survey critically examines the recent literature, concluding that even state-of-the-art architectures and large language models often fall short when probed with relatively simple tasks designed to test basic numerical and arithmetic knowledge.

Analysis of faster matrix multiplication

Since the definition of matrices in 1855, matrix multiplication has played a crucial role in a wide range of fields. Over the years, numerous researchers have dedicated their efforts to improving the time complexity of this fundamental operation. This paper aims to delve into the historical development of matrix multiplication algorithms and methodologies employed to achieve these significant advancements in time complexity. By employing various approaches, researchers have been able to improve the time complexity of matrix multiplication, leading to a significant reduction from O () to O (). Across nearly two centuries, this progress is contributed by a lot of extraordinary scientists and researchers. This paper explores the practical implications of these improvements across various domains, such as computer science, physics, economics, and more. The development of more efficient matrix multiplication algorithms has enabled researchers and practitioners to tackle complex problems and explore new frontiers. In the future, with the rapid growth of machine learning techniques, matrix multiplication will continue to evolve and improve.

Computing Generalized Convolutions Faster Than Brute Force

In this paper, we consider a general notion of convolution. Let D be a finite domain and let D^n be the set of n-length vectors (tuples) of D. Let f :Dtimes Drightarrow D be a function and let oplus _f be a coordinate-wise application of f. The f-Convolution of two functions g,h :D^n rightarrow {-M,ldots ,M} is (g⊛fh)(v):=∑vg,vh∈Dns.t.v=vg⊕fvhg(vg)·h(vh)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\begin{aligned} (g \\mathbin {\\circledast _{f}}h)(\ extbf{v}) {:}{=}\\sum _{\\begin{array}{c} \ extbf{v}_g,\ extbf{v}_h \\in D^n\\\\ \ ext {s.t. } \ extbf{v}= \ extbf{v}_g \\oplus _f \ extbf{v}_h \\end{array}} g(\ extbf{v}_g) \\cdot h(\ extbf{v}_h) \\end{aligned}$$\\end{document}for every textbf{v}in D^n. This problem generalizes many fundamental convolutions such as Subset Convolution, XOR Product, Covering Product or Packing Product, etc. For arbitrary function f and domain D we can compute f-Convolution via brute-force enumeration in widetilde{{mathcal {O}}}(|D|^{2n} cdot textrm{polylog}(M)) time. Our main result is an improvement over this naive algorithm. We show that f-Convolution can be computed exactly in widetilde{{mathcal {O}}}( (c cdot |D|^2)^{n} cdot textrm{polylog}(M)) for constant c {:}{=}3/4 when D has even cardinality. Our main observation is that a cyclic partition of a function f :Dtimes Drightarrow D can be used to speed up the computation of f-Convolution, and we show that an appropriate cyclic partition exists for every f. Furthermore, we demonstrate that a single entry of the f-Convolution can be computed more efficiently. In this variant, we are given two functions g,h :D^n rightarrow {-M,ldots ,M} alongside with a vector textbf{v}in D^n and the task of the f-Query problem is to compute integer (g mathbin {circledast _{f}}h)(textbf{v}). This is a generalization of the well-known Orthogonal Vectors problem. We show that f-Query can be computed in widetilde{{mathcal {O}}}(|D|^{frac{omega }{2} n} cdot textrm{polylog}(M)) time, where omega in [2,2.372) is the exponent of currently fastest matrix multiplication algorithm.

Acceleration of Approximate Matrix Multiplications on GPUs.

Matrix multiplication is important in various information-processing applications, including the computation of eigenvalues and eigenvectors, and in combinatorial optimization algorithms. Therefore, reducing the computation time of matrix products is essential to speed up scientific and practical calculations. Several approaches have been proposed to speed up this process, including GPUs, fast matrix multiplication libraries, custom hardware, and efficient approximate matrix multiplication (AMM) algorithms. However, research to date has yet to focus on accelerating AMMs for general matrices on GPUs, despite the potential of GPUs to perform fast and accurate matrix product calculations. In this paper, we propose a method for improving Monte Carlo AMMs. We also give an analytical solution for the optimal values of the hyperparameters in the proposed method. The proposed method improves the approximation of the matrix product without increasing the computation time compared to the conventional AMMs. It is also designed to work well with parallel operations on GPUs and can be incorporated into various algorithms. Finally, the proposed method is applied to a power method used for eigenvalue computation. We demonstrate that, on an NVIDIA A100 GPU, the computation time can be halved compared to the conventional power method using cuBLAS.

Parallel Matrix Multiplication Algorithms Acquire Connected Network Motifs

The network of interactions between biomolecules is essential to biological processes. Many studies have shown that molecular networks can be analyzed by breaking them down into smaller modules known as network motifs. We hypothesize that identifying the set of possible 5-node motifs and 6-node motifs embedded in a network is a necessary step to elucidate the complex topology of a network. Accomplishing this goal requires determining the complete set of motifs that are composed of five and six connected nodes. 
 We developed a parallel algorithm to reduce time consumption that tackles the exponential problem. It is implemented in matrix multiplication insert in the process of identifying isomorphic patterns and removing the isolated and disconnected patterns. The experiment showed that the parallelization matrix multiplication algorithm is approximately 1.4 times faster than serial programming for identifying 5-node motifs and approximately 1.3 times faster than serial programming for identifying 6-node motifs with all the nodes connected

Finding Matrix Multiplication Algorithms with Classical Planning

Matrix multiplication is a fundamental operation of linear algebra, with applications ranging from quantum physics to artificial intelligence. Given its importance, enormous resources have been invested in the search for faster matrix multiplication algorithms. Recently, this search has been cast as a single-player game. By learning how to play this game efficiently, the newly-introduced AlphaTensor reinforcement learning agent is able to discover many new faster algorithms. In this paper, we show that finding matrix multiplication algorithms can also be cast as a classical planning problem. Based on this observation, we introduce a challenging benchmark suite for classical planning and evaluate state-of-the-art planning techniques on it. We analyze the strengths and limitations of different planning approaches in this domain and show that we can use classical planning to find lower bounds and concrete algorithms for matrix multiplication.

Optimizing massively parallel sparse matrix computing on ARM many-core processor

Sparse matrix multiplication is ubiquitous in many applications such as graph processing and numerical simulation. In recent years, numerous efficient sparse matrix multiplication algorithms and computational libraries have been proposed. However, most of them are oriented to x86 or GPU platforms, while the optimization on ARM many-core platforms has not been well investigated. Our experiments show that existing sparse matrix multiplication libraries for ARM many-core CPU cannot achieve expected parallel performance. Compared with traditional multi-core CPU, ARM many-core CPU has far more cores and often adopts NUMA techniques to scale the memory bandwidth. Its parallel efficiency tends to be restricted by NUMA configuration, memory bandwidth cache contention, etc.In this paper, we propose optimized implementations for sparse matrix computing on ARM many-core CPU. We propose various optimization techniques for several routines of sparse matrix multiplication to ensure coalesced access of matrix elements in the memory. In detail, the optimization techniques include a fine-tuned CSR-based format for ARM architecture, co-optimization of Gustavson’s algorithm with hierarchical cache and dense array strategy to mitigate performance loss caused by handling compressed storage formats. We exploit the coarse-grained NUMA-aware strategy for inter-node parallelism and the fine-grained cache-aware strategy for intra-node parallelism to improve the parallel efficiency of sparse matrix multiplication. The evaluation shows that our implementation consistently outperforms the existing library on ARM many-core processor.

A Comparative Study of Secure Outsourced Matrix Multiplication Based on Homomorphic Encryption

Homomorphic encryption (HE) is a promising solution for handling sensitive data in semi-trusted third-party computing environments, as it enables processing of encrypted data. However, applying sophisticated techniques such as machine learning, statistics, and image processing to encrypted data remains a challenge. The computational complexity of some encrypted operations can significantly increase processing time. In this paper, we focus on the analysis of two state-of-the-art HE matrix multiplication algorithms with the best time and space complexities. We show how their performance depends on the libraries and the execution context, considering the standard Cheon–Kim–Kim–Song (CKKS) HE scheme with fixed-point numbers based on the Microsoft SEAL and PALISADE libraries. We show that Windows OS for the SEAL library and Linux OS for the PALISADE library are the best options. In general, PALISADE-Linux outperforms PALISADE-Windows, SEAL-Linux, and SEAL-Windows by 1.28, 1.59, and 1.67 times on average for different matrix sizes, respectively. We derive high-precision extrapolation formulas to estimate the processing time of HE multiplication of larger matrices.

Development of the application for determination of the optimal partitioning form of large-dimensional matrix’s elements for multiplication on three heterogeneous processors

The article presents the results of the development of a web application for finding the optimal form of splitting matrix elements between three abstract heterogeneous processors when performing the operation of their multiplication. The paper considers five classes of parallel matrix multiplication algorithms: serial communication with a barrier, parallel communication with a barrier, serial communication with overlapping, parallel communication with overlapping, parallel overlapping with alternation. The Hockney model is used to estimate the communication complexity of the algorithms. The work uses six non-rectangular candidate partitioning shapes identified by Ashley DeFlumere in her work [1] as a result of applying the «push» technology of redistribution of matrix elements between the processors: Square Corner, Rectangle Corner, Square Rectangle, Block Rectangle, L-Rectangle, Traditional 1D Rectangular. Determination of the optimality of the form is made on the basis of mathematical models presented in the works [2,3]. The programming languages Python and JavaScript, the Django framework, the pip package manager, and Ajax technology were used to develop a web application. Solving the problem of determining the optimal matrix shape will allow for efficient planning of computing power resources for various scientific fields using parallel matrix multiplication.

A Preliminary Empirical Study of the Power Efficiency of Matrix Multiplication

Matrix multiplication is ubiquitous in high-performance applications. It will be a significant part of exascale workloads where power is a big concern. This work experimentally studied the power efficiency of three matrix multiplication algorithms: the definition-based, Strassen’s divide-and-conquer, and an optimized divide-and-conquer. The study used reliable on-chip integrated voltage regulators for measuring the power. Interactions with memory, mainly cache misses, were thoroughly investigated. The main result was that the optimized divide-and-conquer algorithm, which is the most time-efficient, was also the most power-efficient, but only for cases that fit in the cache. It consumed drastically less overall energy than the other two methods, regardless of placement in memory. For matrix sizes that caused a spill to the main memory, the definition-based algorithm consumes less power than the divide-and-conquer ones at a high total energy cost. The findings from this study may be of interest when cutting power usage is more vital than running for the shortest possible time or least amount of energy.

Weighted slice rank and a minimax correspondence to Strassen's spectra

Structural and computational understanding of tensors is the driving force behind faster matrix multiplication algorithms, the unraveling of quantum entanglement, and the breakthrough on the cap set problem. Strassen's asymptotic spectra program (FOCS 1986) characterizes optimal matrix multiplication algorithms through monotone functionals. Our work advances and makes novel connections among two recent developments in the study of tensors, namely•the slice rank of tensors, a notion of rank for tensors that emerged from the resolution of the cap set problem (Ann. Math. 2017),•and the quantum functionals of tensors (STOC 2018), monotone functionals defined as optimizations over moment polytopes. More precisely, we introduce an extension of slice rank that we call weighted slice rank and we develop a minimax correspondence between the asymptotic weighted slice rank and the quantum functionals. Weighted slice rank encapsulates different notions of bipartiteness of quantum entanglement.The correspondence allows us to give a rank-type characterization of the quantum functionals. Moreover, whereas the original definition of the quantum functionals only works over the complex numbers, this new characterization can be extended to all fields. Thereby, in addition to gaining deeper understanding of Strassen's theory for the complex numbers, we obtain a proposal for quantum functionals over other fields. The finite field case is crucial for combinatorial and algorithmic problems where the field can be optimized over.

Matrix Multiplication in Multiword Arithmetic: Error Analysis and Application to GPU Tensor Cores

In multiword arithmetic, a matrix is represented as the unevaluated sum of two or more lower precision matrices, and a matrix product is formed by multiplying the constituents in low precision. We investigate the use of multiword arithmetic for improving the performance-accuracy tradeoff of matrix multiplication with mixed precision block fused multiply–add (FMA) hardware, focusing especially on the tensor cores available on NVIDIA GPUs. Building on a general block FMA framework, we develop a comprehensive error analysis of multiword matrix multiplication. After confirming the theoretical error bounds experimentally by simulating low precision in software, we use the cuBLAS and CUTLASS libraries to implement a number of matrix multiplication algorithms using double-fp16 (double-binary16) arithmetic. When running the algorithms on NVIDIA V100 and A100 GPUs, we find that double-fp16 is not as accurate as fp32 (binary32) arithmetic despite satisfying the same worst-case error bound. Using probabilistic error analysis, we explain why this issue is likely to be caused by the rounding mode used by the NVIDIA tensor cores, and we propose a parameterized blocked summation algorithm that alleviates the problem and significantly improves the performance-accuracy tradeoff.

Privacy-preserving cancer type prediction with homomorphic encryption

Cancer genomics tailors diagnosis and treatment based on an individual’s genetic information and is the crux of precision medicine. However, analysis and maintenance of high volume of genetic mutation data to build a machine learning (ML) model to predict the cancer type is a computationally expensive task and is often outsourced to powerful cloud servers, raising critical privacy concerns for patients’ data. Homomorphic encryption (HE) enables computation on encrypted data, thus, providing cryptographic guarantees to protect privacy. But restrictive overheads of encrypted computation deter its usage. In this work, we explore the challenges of privacy preserving cancer type prediction using a dataset consisting of more than 2 million genetic mutations from 2713 patients for several cancer types by building a highly accurate ML model and then implementing its privacy preserving version in HE. Our solution for cancer type inference encodes somatic mutations based on their impact on the cancer genomes into the feature space and then uses statistical tests for feature selection. We propose a fast matrix multiplication algorithm for HE-based model. Our final model achieves 0.98 micro-average area under curve improving accuracy from 70.08 to 83.61% , being 550 times faster than the standard matrix multiplication-based privacy-preserving models. Our tool can be found at https://github.com/momalab/octal-candet.

Optimal sampling algorithms for block matrix multiplication

In this paper, we investigate the randomized algorithms for block matrix multiplication from random sampling perspective. Specifically, based on the A-optimal design criterion, we obtain the optimal sampling probabilities and sampling block sizes. To improve the practicability of the block sizes, two modified ones with less computation cost are provided. With respect to the second modified block size, we devise a two step algorithm. Moreover, the probability error bounds for the proposed algorithms are also given. Extensive numerical results show that our methods outperform the state-of-the-art ones given in the literature.

Binary operations on neuromorphic hardware with application to linear algebraic operations and stochastic equations

Non-von Neumann computational hardware, based on neuron-inspired, non-linear elements connected via linear, weighted synapses—so-called neuromorphic systems—is a viable computational substrate. Since neuromorphic systems have been shown to use less power than CPUs for many applications, they are of potential use in autonomous systems such as robots, drones, and satellites, for which power resources are at a premium. The power used by neuromorphic systems is approximately proportional to the number of spiking events produced by neurons on-chip. However, typical information encoding on these chips is in the form of firing rates that unarily encode information. That is, the number of spikes generated by a neuron is meant to be proportional to an encoded value used in a computation or algorithm. Unary encoding is less efficient (produces more spikes) than binary encoding. For this reason, here we present neuromorphic computational mechanisms for implementing binary two’s complement operations. We use the mechanisms to construct a neuromorphic, binary matrix multiplication algorithm that may be used as a primitive for linear differential equation integration, deep networks, and other standard calculations. We also construct a random walk circuit and apply it in Brownian motion simulations. We study how both algorithms scale in circuit size and iteration time.

ASIC Implementation of Bit Matrix Multiplier

In computer science and digital electronics, a bit matrix multiplier (BMM) is a mathematical operation that is used to quickly multiply binary matrices. BMM is a basic component of many computer algorithms and is utilized in fields including machine learning, image processing, and cryptography. BMM creates a new matrix that represents the product of the two input matrices by performing logical AND and XOR operations on each matrix element’s binary value. BMM is a crucial method for large-scale matrix operations since it has a lower computational complexity than conventional matrix multiplication. Reduced computational complexity: When compared to conventional matrix multiplication algorithms, BMM has a lower computational complexity since it performs matrix multiplication using bitwise operations like logical AND and XOR. Faster processing speeds are the result, particularly for complex matrix computations. Less memory is needed to store the binary values of the matrices in BMM because these values can be expressed using Boolean logic. As a result, less memory is needed, and the resources can be used more effectively.

Discovering faster matrix multiplication algorithms with reinforcement learning

Improving the efficiency of algorithms for fundamental computations can have a widespread impact, as it can affect the overall speed of a large amount of computations. Matrix multiplication is one such primitive task, occurring in many systems—from neural networks to scientific computing routines. The automatic discovery of algorithms using machine learning offers the prospect of reaching beyond human intuition and outperforming the current best human-designed algorithms. However, automating the algorithm discovery procedure is intricate, as the space of possible algorithms is enormous. Here we report a deep reinforcement learning approach based on AlphaZero1 for discovering efficient and provably correct algorithms for the multiplication of arbitrary matrices. Our agent, AlphaTensor, is trained to play a single-player game where the objective is finding tensor decompositions within a finite factor space. AlphaTensor discovered algorithms that outperform the state-of-the-art complexity for many matrix sizes. Particularly relevant is the case of 4 × 4 matrices in a finite field, where AlphaTensor’s algorithm improves on Strassen’s two-level algorithm for the first time, to our knowledge, since its discovery 50 years ago2. We further showcase the flexibility of AlphaTensor through different use-cases: algorithms with state-of-the-art complexity for structured matrix multiplication and improved practical efficiency by optimizing matrix multiplication for runtime on specific hardware. Our results highlight AlphaTensor’s ability to accelerate the process of algorithmic discovery on a range of problems, and to optimize for different criteria.

Ultra-fast and efficient implementation schemes of complex matrix multiplication algorithm for VLIW architectures

The Complex Matrix Multiplication (CMM) algorithm is known to require a high computing performance and presenting exceptional challenges in real-life applications. Recent advances in Very Long Instruction Word (VLIW) based Digital Signal Processors (DSP) demonstrated high computing capabilities with a very low power consumption. In this work, we propose three ultra-fast, parallel and efficient VLIW implementation approaches of the CMM algorithm which could be used to meet tighter real-time constraints of several signal and image processing applications like radars. A novel parallel kernel, task mapping strategy and low-level optimization techniques are suggested, to fit a set of modern VLIW architectures. Additionally, an original memory access management technique was adopted to accelerate the algorithm by avoiding cache misses and bank conflicts. The experimental results showed the effectiveness of the proposed approaches where a peak performance of 15.89 GFLOPS was achieved on one C66x DSP core with a core utilization of 99% and a speedup of about 1.61, 3 and 10 compared to the state-of-the-art, the most optimized vendor and the conventional approaches, respectively.