Matrix Multiplication Algorithm Research Articles

Semi-analytical solution of Brent equations

A parametrization of Brent equations is proposed which admit for a several times reduction of the number of unknowns and equations. The arising equations are solved numerically, and for the resulting fast matrix multiplication algorithms many known values of rank are reproduced and even improved, in particular, the designs (4,4,4;48) and (2,4,5;32) are found.

Extension of accurate numerical algorithms for matrix multiplication based on error-free transformation

Extension of accurate numerical algorithms for matrix multiplication based on error-free transformation

Correction of ‘J. Laderman, V. Pan, X.–H. Sha, On practical Algorithms for Accelerated Matrix Multiplication, Linear Algebra and its Applications. Vol. 162-164 (1992) pp. 557-588’

In this article, we corrected the trilinear formula for triple disjoint matrix multiplication given in the article ‘J. Laderman, V. Pan, X. H. Sha, On practical Algorithms for Accelerated Matrix Multiplication, Linear Algebra and its Applications. Vol. 162-164 (1992) pp. 557-588’, which is incorrect for matrix dimensions equal to two or greater. That formula is a base of two algorithms, for disjoint and single matrix multiplication. The necessary correction made the amount of non–scalar products rise, slightly increasing the algorithm time complexity. We also corrected explicit formulas, in the bilinear form, for triple disjoint matrix multiplication. They are explicit, and thus convenient for the practical use of the fast matrix multiplication algorithms in question.

Algorithms for Matrix Multiplication via Sampling and Opportunistic Matrix Multiplication

Algorithms for Matrix Multiplication via Sampling and Opportunistic Matrix Multiplication

Improving compressed matrix multiplication using control variate method

The seminal work by Pagh [1] proposed a matrix multiplication algorithm for real-valued squared matrices called Compressed Matrix Multiplication (CMM) having a sparse matrix output product. The algorithm is based on a popular sketching technique called Count-Sketch [2] and Fast Fourier Transform (FFT). For input square matrices A and B of order n and the product matrix AB with Frobenius norm ||AB||F, the algorithm offers an unbiased estimate for each entry, i.e., (AB)i,j of the product matrix AB with a variance bounded by ||AB||F2/b, where b is the compressed bucket size. Thus, the variance will eventually become high for a small bucket size. In this work, we address the high variance problem of CMM with the help of a simple and practical technique based on classical variance reduction methods in statistics. Our techniques rely on the Control Variate (CV) method. We suggest rigorous theoretical analysis for variance reduction and complement it via supporting empirical evidence.

Investigation of Energy and Power Characteristics of Various Matrix Multiplication Algorithms

Investigation of Energy and Power Characteristics of Various Matrix Multiplication Algorithms

Research on fast matrix multiplication algorithm

This research mainly focuses on fast matrix multiplication algorithms. Fast matrix multiplication is one of the most fundamental problems in computer science. The fast matrix multiplication algorithm differs from conventional matrix multiplication in that it offers a faster computational approach that can perform the operation in less than O(n3) time complexity. This algorithm provides a more efficient method for multiplying matrices, significantly reducing the computational requirements. The Laser method, developed by Coppersmith and Winograd, is an algorithm for matrix multiplication that does not involve direct computation. It establishes a relationship between matrix multiplication and tensors and simplifies the operation by finding an intermediate tensor that is computationally manageable. This method applies a series of simplification operations to determine an upper bound on the computational complexity of matrix multiplication. However, as matrices become larger, the computational and memory requirements increase, posing challenges for practical implementation. This research will present the main ideas and performance of the Laser method and discuss the improvements made to the Laser method, including refined analysis and asymmetric hashing techniques. Additionally, it highlights the need for further exploration, such as parallel computing and optimization strategies, to enhance the efficiency of matrix multiplication algorithms. Furthermore, this research will also provide a prospectus for the future of matrix multiplication algorithms, such as the practical implementation of the Laser method.

Can Neural Networks Do Arithmetic? A Survey on the Elementary Numerical Skills of State-of-the-Art Deep Learning Models

Creating learning models that can exhibit sophisticated reasoning abilities is one of the greatest challenges in deep learning research, and mathematics is rapidly becoming one of the target domains for assessing scientific progress in this direction. In the past few years there has been an explosion of neural network architectures, datasets, and benchmarks specifically designed to tackle mathematical problems, reporting impressive achievements in disparate fields such as automated theorem proving, numerical integration, and the discovery of new conjectures or matrix multiplication algorithms. However, despite this notable success it is still unclear whether deep learning models possess an elementary understanding of quantities and numbers. This survey critically examines the recent literature, concluding that even state-of-the-art architectures and large language models often fall short when probed with relatively simple tasks designed to test basic numerical and arithmetic knowledge.

Analysis of faster matrix multiplication

Since the definition of matrices in 1855, matrix multiplication has played a crucial role in a wide range of fields. Over the years, numerous researchers have dedicated their efforts to improving the time complexity of this fundamental operation. This paper aims to delve into the historical development of matrix multiplication algorithms and methodologies employed to achieve these significant advancements in time complexity. By employing various approaches, researchers have been able to improve the time complexity of matrix multiplication, leading to a significant reduction from O () to O (). Across nearly two centuries, this progress is contributed by a lot of extraordinary scientists and researchers. This paper explores the practical implications of these improvements across various domains, such as computer science, physics, economics, and more. The development of more efficient matrix multiplication algorithms has enabled researchers and practitioners to tackle complex problems and explore new frontiers. In the future, with the rapid growth of machine learning techniques, matrix multiplication will continue to evolve and improve.

Computing Generalized Convolutions Faster Than Brute Force

In this paper, we consider a general notion of convolution. Let D be a finite domain and let D^n be the set of n-length vectors (tuples) of D. Let f :Dtimes Drightarrow D be a function and let oplus _f be a coordinate-wise application of f. The f-Convolution of two functions g,h :D^n rightarrow {-M,ldots ,M} is (g⊛fh)(v):=∑vg,vh∈Dns.t.v=vg⊕fvhg(vg)·h(vh)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\begin{aligned} (g \\mathbin {\\circledast _{f}}h)(\ extbf{v}) {:}{=}\\sum _{\\begin{array}{c} \ extbf{v}_g,\ extbf{v}_h \\in D^n\\\\ \ ext {s.t. } \ extbf{v}= \ extbf{v}_g \\oplus _f \ extbf{v}_h \\end{array}} g(\ extbf{v}_g) \\cdot h(\ extbf{v}_h) \\end{aligned}$$\\end{document}for every textbf{v}in D^n. This problem generalizes many fundamental convolutions such as Subset Convolution, XOR Product, Covering Product or Packing Product, etc. For arbitrary function f and domain D we can compute f-Convolution via brute-force enumeration in widetilde{{mathcal {O}}}(|D|^{2n} cdot textrm{polylog}(M)) time. Our main result is an improvement over this naive algorithm. We show that f-Convolution can be computed exactly in widetilde{{mathcal {O}}}( (c cdot |D|^2)^{n} cdot textrm{polylog}(M)) for constant c {:}{=}3/4 when D has even cardinality. Our main observation is that a cyclic partition of a function f :Dtimes Drightarrow D can be used to speed up the computation of f-Convolution, and we show that an appropriate cyclic partition exists for every f. Furthermore, we demonstrate that a single entry of the f-Convolution can be computed more efficiently. In this variant, we are given two functions g,h :D^n rightarrow {-M,ldots ,M} alongside with a vector textbf{v}in D^n and the task of the f-Query problem is to compute integer (g mathbin {circledast _{f}}h)(textbf{v}). This is a generalization of the well-known Orthogonal Vectors problem. We show that f-Query can be computed in widetilde{{mathcal {O}}}(|D|^{frac{omega }{2} n} cdot textrm{polylog}(M)) time, where omega in [2,2.372) is the exponent of currently fastest matrix multiplication algorithm.

Acceleration of Approximate Matrix Multiplications on GPUs.

Matrix multiplication is important in various information-processing applications, including the computation of eigenvalues and eigenvectors, and in combinatorial optimization algorithms. Therefore, reducing the computation time of matrix products is essential to speed up scientific and practical calculations. Several approaches have been proposed to speed up this process, including GPUs, fast matrix multiplication libraries, custom hardware, and efficient approximate matrix multiplication (AMM) algorithms. However, research to date has yet to focus on accelerating AMMs for general matrices on GPUs, despite the potential of GPUs to perform fast and accurate matrix product calculations. In this paper, we propose a method for improving Monte Carlo AMMs. We also give an analytical solution for the optimal values of the hyperparameters in the proposed method. The proposed method improves the approximation of the matrix product without increasing the computation time compared to the conventional AMMs. It is also designed to work well with parallel operations on GPUs and can be incorporated into various algorithms. Finally, the proposed method is applied to a power method used for eigenvalue computation. We demonstrate that, on an NVIDIA A100 GPU, the computation time can be halved compared to the conventional power method using cuBLAS.

Parallel Matrix Multiplication Algorithms Acquire Connected Network Motifs

The network of interactions between biomolecules is essential to biological processes. Many studies have shown that molecular networks can be analyzed by breaking them down into smaller modules known as network motifs. We hypothesize that identifying the set of possible 5-node motifs and 6-node motifs embedded in a network is a necessary step to elucidate the complex topology of a network. Accomplishing this goal requires determining the complete set of motifs that are composed of five and six connected nodes. 
 We developed a parallel algorithm to reduce time consumption that tackles the exponential problem. It is implemented in matrix multiplication insert in the process of identifying isomorphic patterns and removing the isolated and disconnected patterns. The experiment showed that the parallelization matrix multiplication algorithm is approximately 1.4 times faster than serial programming for identifying 5-node motifs and approximately 1.3 times faster than serial programming for identifying 6-node motifs with all the nodes connected

Finding Matrix Multiplication Algorithms with Classical Planning

Matrix multiplication is a fundamental operation of linear algebra, with applications ranging from quantum physics to artificial intelligence. Given its importance, enormous resources have been invested in the search for faster matrix multiplication algorithms. Recently, this search has been cast as a single-player game. By learning how to play this game efficiently, the newly-introduced AlphaTensor reinforcement learning agent is able to discover many new faster algorithms. In this paper, we show that finding matrix multiplication algorithms can also be cast as a classical planning problem. Based on this observation, we introduce a challenging benchmark suite for classical planning and evaluate state-of-the-art planning techniques on it. We analyze the strengths and limitations of different planning approaches in this domain and show that we can use classical planning to find lower bounds and concrete algorithms for matrix multiplication.

Optimizing massively parallel sparse matrix computing on ARM many-core processor

Sparse matrix multiplication is ubiquitous in many applications such as graph processing and numerical simulation. In recent years, numerous efficient sparse matrix multiplication algorithms and computational libraries have been proposed. However, most of them are oriented to x86 or GPU platforms, while the optimization on ARM many-core platforms has not been well investigated. Our experiments show that existing sparse matrix multiplication libraries for ARM many-core CPU cannot achieve expected parallel performance. Compared with traditional multi-core CPU, ARM many-core CPU has far more cores and often adopts NUMA techniques to scale the memory bandwidth. Its parallel efficiency tends to be restricted by NUMA configuration, memory bandwidth cache contention, etc.In this paper, we propose optimized implementations for sparse matrix computing on ARM many-core CPU. We propose various optimization techniques for several routines of sparse matrix multiplication to ensure coalesced access of matrix elements in the memory. In detail, the optimization techniques include a fine-tuned CSR-based format for ARM architecture, co-optimization of Gustavson’s algorithm with hierarchical cache and dense array strategy to mitigate performance loss caused by handling compressed storage formats. We exploit the coarse-grained NUMA-aware strategy for inter-node parallelism and the fine-grained cache-aware strategy for intra-node parallelism to improve the parallel efficiency of sparse matrix multiplication. The evaluation shows that our implementation consistently outperforms the existing library on ARM many-core processor.

A Comparative Study of Secure Outsourced Matrix Multiplication Based on Homomorphic Encryption

Homomorphic encryption (HE) is a promising solution for handling sensitive data in semi-trusted third-party computing environments, as it enables processing of encrypted data. However, applying sophisticated techniques such as machine learning, statistics, and image processing to encrypted data remains a challenge. The computational complexity of some encrypted operations can significantly increase processing time. In this paper, we focus on the analysis of two state-of-the-art HE matrix multiplication algorithms with the best time and space complexities. We show how their performance depends on the libraries and the execution context, considering the standard Cheon–Kim–Kim–Song (CKKS) HE scheme with fixed-point numbers based on the Microsoft SEAL and PALISADE libraries. We show that Windows OS for the SEAL library and Linux OS for the PALISADE library are the best options. In general, PALISADE-Linux outperforms PALISADE-Windows, SEAL-Linux, and SEAL-Windows by 1.28, 1.59, and 1.67 times on average for different matrix sizes, respectively. We derive high-precision extrapolation formulas to estimate the processing time of HE multiplication of larger matrices.

Development of the application for determination of the optimal partitioning form of large-dimensional matrix’s elements for multiplication on three heterogeneous processors

The article presents the results of the development of a web application for finding the optimal form of splitting matrix elements between three abstract heterogeneous processors when performing the operation of their multiplication. The paper considers five classes of parallel matrix multiplication algorithms: serial communication with a barrier, parallel communication with a barrier, serial communication with overlapping, parallel communication with overlapping, parallel overlapping with alternation. The Hockney model is used to estimate the communication complexity of the algorithms. The work uses six non-rectangular candidate partitioning shapes identified by Ashley DeFlumere in her work [1] as a result of applying the «push» technology of redistribution of matrix elements between the processors: Square Corner, Rectangle Corner, Square Rectangle, Block Rectangle, L-Rectangle, Traditional 1D Rectangular. Determination of the optimality of the form is made on the basis of mathematical models presented in the works [2,3]. The programming languages Python and JavaScript, the Django framework, the pip package manager, and Ajax technology were used to develop a web application. Solving the problem of determining the optimal matrix shape will allow for efficient planning of computing power resources for various scientific fields using parallel matrix multiplication.

A Preliminary Empirical Study of the Power Efficiency of Matrix Multiplication

Matrix multiplication is ubiquitous in high-performance applications. It will be a significant part of exascale workloads where power is a big concern. This work experimentally studied the power efficiency of three matrix multiplication algorithms: the definition-based, Strassen’s divide-and-conquer, and an optimized divide-and-conquer. The study used reliable on-chip integrated voltage regulators for measuring the power. Interactions with memory, mainly cache misses, were thoroughly investigated. The main result was that the optimized divide-and-conquer algorithm, which is the most time-efficient, was also the most power-efficient, but only for cases that fit in the cache. It consumed drastically less overall energy than the other two methods, regardless of placement in memory. For matrix sizes that caused a spill to the main memory, the definition-based algorithm consumes less power than the divide-and-conquer ones at a high total energy cost. The findings from this study may be of interest when cutting power usage is more vital than running for the shortest possible time or least amount of energy.

Weighted slice rank and a minimax correspondence to Strassen's spectra

Structural and computational understanding of tensors is the driving force behind faster matrix multiplication algorithms, the unraveling of quantum entanglement, and the breakthrough on the cap set problem. Strassen's asymptotic spectra program (FOCS 1986) characterizes optimal matrix multiplication algorithms through monotone functionals. Our work advances and makes novel connections among two recent developments in the study of tensors, namely•the slice rank of tensors, a notion of rank for tensors that emerged from the resolution of the cap set problem (Ann. Math. 2017),•and the quantum functionals of tensors (STOC 2018), monotone functionals defined as optimizations over moment polytopes. More precisely, we introduce an extension of slice rank that we call weighted slice rank and we develop a minimax correspondence between the asymptotic weighted slice rank and the quantum functionals. Weighted slice rank encapsulates different notions of bipartiteness of quantum entanglement.The correspondence allows us to give a rank-type characterization of the quantum functionals. Moreover, whereas the original definition of the quantum functionals only works over the complex numbers, this new characterization can be extended to all fields. Thereby, in addition to gaining deeper understanding of Strassen's theory for the complex numbers, we obtain a proposal for quantum functionals over other fields. The finite field case is crucial for combinatorial and algorithmic problems where the field can be optimized over.

Matrix Multiplication in Multiword Arithmetic: Error Analysis and Application to GPU Tensor Cores

In multiword arithmetic, a matrix is represented as the unevaluated sum of two or more lower precision matrices, and a matrix product is formed by multiplying the constituents in low precision. We investigate the use of multiword arithmetic for improving the performance-accuracy tradeoff of matrix multiplication with mixed precision block fused multiply–add (FMA) hardware, focusing especially on the tensor cores available on NVIDIA GPUs. Building on a general block FMA framework, we develop a comprehensive error analysis of multiword matrix multiplication. After confirming the theoretical error bounds experimentally by simulating low precision in software, we use the cuBLAS and CUTLASS libraries to implement a number of matrix multiplication algorithms using double-fp16 (double-binary16) arithmetic. When running the algorithms on NVIDIA V100 and A100 GPUs, we find that double-fp16 is not as accurate as fp32 (binary32) arithmetic despite satisfying the same worst-case error bound. Using probabilistic error analysis, we explain why this issue is likely to be caused by the rounding mode used by the NVIDIA tensor cores, and we propose a parameterized blocked summation algorithm that alleviates the problem and significantly improves the performance-accuracy tradeoff.

Privacy-preserving cancer type prediction with homomorphic encryption

Cancer genomics tailors diagnosis and treatment based on an individual’s genetic information and is the crux of precision medicine. However, analysis and maintenance of high volume of genetic mutation data to build a machine learning (ML) model to predict the cancer type is a computationally expensive task and is often outsourced to powerful cloud servers, raising critical privacy concerns for patients’ data. Homomorphic encryption (HE) enables computation on encrypted data, thus, providing cryptographic guarantees to protect privacy. But restrictive overheads of encrypted computation deter its usage. In this work, we explore the challenges of privacy preserving cancer type prediction using a dataset consisting of more than 2 million genetic mutations from 2713 patients for several cancer types by building a highly accurate ML model and then implementing its privacy preserving version in HE. Our solution for cancer type inference encodes somatic mutations based on their impact on the cancer genomes into the feature space and then uses statistical tests for feature selection. We propose a fast matrix multiplication algorithm for HE-based model. Our final model achieves 0.98 micro-average area under curve improving accuracy from 70.08 to 83.61% , being 550 times faster than the standard matrix multiplication-based privacy-preserving models. Our tool can be found at https://github.com/momalab/octal-candet.