Matrix Multiplication Algorithm Research Articles

Optimal sampling algorithms for block matrix multiplication

In this paper, we investigate the randomized algorithms for block matrix multiplication from random sampling perspective. Specifically, based on the A-optimal design criterion, we obtain the optimal sampling probabilities and sampling block sizes. To improve the practicability of the block sizes, two modified ones with less computation cost are provided. With respect to the second modified block size, we devise a two step algorithm. Moreover, the probability error bounds for the proposed algorithms are also given. Extensive numerical results show that our methods outperform the state-of-the-art ones given in the literature.

Binary operations on neuromorphic hardware with application to linear algebraic operations and stochastic equations

Non-von Neumann computational hardware, based on neuron-inspired, non-linear elements connected via linear, weighted synapses—so-called neuromorphic systems—is a viable computational substrate. Since neuromorphic systems have been shown to use less power than CPUs for many applications, they are of potential use in autonomous systems such as robots, drones, and satellites, for which power resources are at a premium. The power used by neuromorphic systems is approximately proportional to the number of spiking events produced by neurons on-chip. However, typical information encoding on these chips is in the form of firing rates that unarily encode information. That is, the number of spikes generated by a neuron is meant to be proportional to an encoded value used in a computation or algorithm. Unary encoding is less efficient (produces more spikes) than binary encoding. For this reason, here we present neuromorphic computational mechanisms for implementing binary two’s complement operations. We use the mechanisms to construct a neuromorphic, binary matrix multiplication algorithm that may be used as a primitive for linear differential equation integration, deep networks, and other standard calculations. We also construct a random walk circuit and apply it in Brownian motion simulations. We study how both algorithms scale in circuit size and iteration time.

ASIC Implementation of Bit Matrix Multiplier

In computer science and digital electronics, a bit matrix multiplier (BMM) is a mathematical operation that is used to quickly multiply binary matrices. BMM is a basic component of many computer algorithms and is utilized in fields including machine learning, image processing, and cryptography. BMM creates a new matrix that represents the product of the two input matrices by performing logical AND and XOR operations on each matrix element’s binary value. BMM is a crucial method for large-scale matrix operations since it has a lower computational complexity than conventional matrix multiplication. Reduced computational complexity: When compared to conventional matrix multiplication algorithms, BMM has a lower computational complexity since it performs matrix multiplication using bitwise operations like logical AND and XOR. Faster processing speeds are the result, particularly for complex matrix computations. Less memory is needed to store the binary values of the matrices in BMM because these values can be expressed using Boolean logic. As a result, less memory is needed, and the resources can be used more effectively.

Discovering faster matrix multiplication algorithms with reinforcement learning

Improving the efficiency of algorithms for fundamental computations can have a widespread impact, as it can affect the overall speed of a large amount of computations. Matrix multiplication is one such primitive task, occurring in many systems—from neural networks to scientific computing routines. The automatic discovery of algorithms using machine learning offers the prospect of reaching beyond human intuition and outperforming the current best human-designed algorithms. However, automating the algorithm discovery procedure is intricate, as the space of possible algorithms is enormous. Here we report a deep reinforcement learning approach based on AlphaZero1 for discovering efficient and provably correct algorithms for the multiplication of arbitrary matrices. Our agent, AlphaTensor, is trained to play a single-player game where the objective is finding tensor decompositions within a finite factor space. AlphaTensor discovered algorithms that outperform the state-of-the-art complexity for many matrix sizes. Particularly relevant is the case of 4 × 4 matrices in a finite field, where AlphaTensor’s algorithm improves on Strassen’s two-level algorithm for the first time, to our knowledge, since its discovery 50 years ago2. We further showcase the flexibility of AlphaTensor through different use-cases: algorithms with state-of-the-art complexity for structured matrix multiplication and improved practical efficiency by optimizing matrix multiplication for runtime on specific hardware. Our results highlight AlphaTensor’s ability to accelerate the process of algorithmic discovery on a range of problems, and to optimize for different criteria.

Ultra-fast and efficient implementation schemes of complex matrix multiplication algorithm for VLIW architectures

The Complex Matrix Multiplication (CMM) algorithm is known to require a high computing performance and presenting exceptional challenges in real-life applications. Recent advances in Very Long Instruction Word (VLIW) based Digital Signal Processors (DSP) demonstrated high computing capabilities with a very low power consumption. In this work, we propose three ultra-fast, parallel and efficient VLIW implementation approaches of the CMM algorithm which could be used to meet tighter real-time constraints of several signal and image processing applications like radars. A novel parallel kernel, task mapping strategy and low-level optimization techniques are suggested, to fit a set of modern VLIW architectures. Additionally, an original memory access management technique was adopted to accelerate the algorithm by avoiding cache misses and bank conflicts. The experimental results showed the effectiveness of the proposed approaches where a peak performance of 15.89 GFLOPS was achieved on one C66x DSP core with a core utilization of 99% and a speedup of about 1.61, 3 and 10 compared to the state-of-the-art, the most optimized vendor and the conventional approaches, respectively.

Large-scale distributed linear algebra with tensor processing units

We have repurposed Google tensor processing units (TPUs), application-specific chips developed for machine learning, into large-scale dense linear algebra supercomputers. The TPUs' fast intercore interconnects (ICIs), physically two-dimensional network topology, and high-bandwidth memory (HBM) permit distributed matrix multiplication algorithms to rapidly become computationally bound. In this regime, the matrix-multiply units (MXUs) dominate the runtime, yielding impressive scaling, performance, and raw size: Operating in float32 precision, a full 2,048-core pod of third-generation TPUs can multiply two matrices with linear size [Formula: see text] in about 2 min. Via curated algorithms emphasizing large, single-core matrix multiplications, other tasks in dense linear algebra can similarly scale. As examples, we present 1) QR decomposition; 2) resolution of linear systems; and 3) the computation of matrix functions by polynomial iteration, demonstrated by the matrix polar factorization.

FCNNLib: A Flexible Convolution Algorithm Library for Deep Learning on FPGAs

Convolution features huge complexity and demands high computation capability. Among hardware platforms, field programmable gate array (FPGA) emerges as a promising solution for its substantial available parallelism and energy efficiency. Besides, convolution can be implemented with different algorithms, including conventional, general matrix–matrix multiplication (GEMM), Winograd, and fast Fourier transformation (FFT) algorithms, which are diverse in arithmetic complexity, resource requirement, etc. Different convolutional neural network (CNN) models have different topologies and structures, favoring different convolution algorithms. In response, software libraries such as cuDNN provide a variety of computational primitives to support these algorithms. However, supporting such libraries on FPGAs is challenging. First, multiple algorithms can share the FPGA resources spatially as well as temporally, introducing either reconfiguration overhead or resource underutilization. Second, FPGA implementation remains a significant challenge for library developers. It typically requires significant specialized hardware knowledge. In this article, we propose <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FCNNLib</monospace> , an efficient and scalable convolution algorithm library on FPGAs. To coordinate multiple convolution algorithms on FPGAs, we develop three schedulings: 1) spatial; 2) temporal; and 3) hybrid, which exhibit different tradeoffs in latency and throughput. We explore these schedulings by balancing the reconfiguration overhead, resource utilization, and optimization objectives of the CNNs. Then, we provide efficient and tunable algorithm templates that allow performance tuning through performance and resource models. To arm the users, <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FCNNLib</monospace> exposes a set of interfaces to support high-level application designs. We demonstrate the usability of <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FCNNLib</monospace> with state-of-the-art CNNs. <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FCNNLib</monospace> achieves up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$44.6\times $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.76\times $ </tex-math></inline-formula> energy efficiency in various scenarios compared with software libraries for CPUs and GPUs, respectively.

Stark: Fast and Scalable Strassen’s Matrix Multiplication Using Apache Spark

This article presents a new fast, highly scalable distributed matrix multiplication algorithm on Apache Spark, called <i>Stark</i> , based on Strassen’s matrix multiplication algorithm. Stark preserves Strassen’s seven multiplications scheme in a distributed environment and thus achieves asymptotically faster execution time. It creates a distributed recursion tree of computation where each level of the tree corresponds to division and combination of distributed matrix blocks stored in the form of Resilient Distributed Datasets (RDDs). It processes each divide and combine step in parallel and memorises the sub-matrices by intelligently tagging matrix blocks in it. To the best of our knowledge, Stark is the first implementation of a distribute Strassen’s algorithm on Spark platform. We also report a detailed complexity analysis for the proposed algorithm, taking into account computation and communication costs. Experimental results suggest that Stark outperforms existing distributed matrix multiplication implementations on Spark – <i>Marlin</i> and <i>MLLib</i> , for high matrix sizes ( <inline-formula><tex-math notation="LaTeX">$\geq 16384\times 16384$</tex-math></inline-formula> ). Our experiments reveal optimal block sizes for each matrix size, which is also shown from theoretical analysis. We also show that the experimental and theoretical running times for Stark match closely. It has also been shown experimentally that Stark exhibits strong scalability with increasing number of executors.

C-Lop: Accurate contention-based modeling of MPI concurrent communication

MPI communication optimization is a crucial stage to optimize high-performance applications. As a formal analysis of MPI communication, the communication performance models have made some achievements in improving the efficiency of collective algorithms and optimizing communication scheduling. However, previous models are difficult to model asynchronous concurrent communication and do not take into account numerous contention factors. In this paper, we present C-Lop, an incremental MPI performance model based on τ-Lop. Firstly, C-Lop proposes a method for asynchronous modeling of concurrent communication. As the only model that considers concurrent transmission, τ-Lop describes the cost of the all processes as a whole without distinguishing the cost of each process. Here, C-Lop uses the idea of asynchronous modeling that describe the cost of the system by averaging the communication cost per process. It can describe the communication cost for some systems with out-of-sync communication more accurately. Moreover, C-Lop introduces the parameter C to represent the contention, and considers the contention of concurrent transmissions on network-on-chip, data reuse, and contention of noncommunication processes to make a more accuracy estimation. Furthermore, parameter C can be customized to fit more application scenarios. In addition, we evaluate several common collective algorithms, a matrix multiplication algorithm (SUMMA), and two kinds of communication in a three-dimensional multi-grid application on the Tianhe-3 prototype, and results show that C-Lop outperforms the competition.

LogSC: Model-based one-sided communication performance estimation

One-sided communication (also known as remote memory access, or RMA) in the Message Passing Interface (MPI) is a communication interface that has been introduced in MPI-2 (1997) that enables new more efficient programming models. In MPI-3, some new more flexible and efficient primitives have been introduced, which makes it easier to use and more deployable.However, compared to traditional two-sided communication, little work has been performed on analyzing one-sided communication costs, which urgently requires formal analysis. The communication performance model is a formal analysis of communication and its cost. We focus on the software model, whose core idea is that the transmission can be represented as a sequence of implicit transfers and data movements. This approach is a suitable solution for concurrent communication modeling.We propose LogSC, which consists of the window cost, transmission cost, synchronization cost, and computational cost in atomic operations. In this paper, LogSC is used to model most of the operations in one-sided communication, including the put/get operation, atomic operation, and shared memory programming of MPI. We model and evaluate the parallel tests of IMB, collectives designed by combining MPI and the MPI shared memory (MPI+MPI), and the communication in the scalable universal matrix multiplication algorithm (SUMMA), which is a common matrix multiplication algorithm. Experiments show that our modeling has high accuracy, which makes up for the lack of existing models.

PVD-FL: A Privacy-Preserving and Verifiable Decentralized Federated Learning Framework

Over the past years, the increasingly severe data island problem has spawned an emerging distributed deep learning framework—federated learning, in which the global model can be constructed over multiple participants without directly sharing their raw data. Despite its promising prospect, there are still many security challenges in federated learning, such as privacy preservation and integrity verification. Furthermore, federated learning is usually performed with the assistance of a center, which is prone to cause trust worries and communicational bottlenecks. To tackle these challenges, in this paper, we propose a privacy-preserving and verifiable decentralized federated learning framework, named PVD-FL, which can achieve secure deep learning model training under a decentralized architecture. Specifically, we first design an efficient and verifiable cipher-based matrix multiplication (EVCM) algorithm to execute the most basic calculation in deep learning. Then, by employing EVCM, we design a suite of decentralized algorithms to construct the PVD-FL framework, which ensures the confidentiality of both global model and local update and the verification of every training step. Detailed security analysis shows that PVD-FL can well protect privacy against various inference attacks and guarantee training integrity. In addition, the extensive experiments on real-world datasets also demonstrate that PVD-FL can achieve lossless accuracy and practical performance.

Linear‐time algorithms for eliminating claws in graphs

AbstractSince many ‐complete graph problems are polynomial‐time solvable when restricted to claw‐free graphs, we study the problem of determining the distance of a given graph to a claw‐free graph, considering vertex elimination a measure. Claw‐free Vertex Deletion (CFVD) consists of determining the minimum number of vertices to be removed from a graph such that the resulting graph is claw‐free. Although CFVD is ‐hard in general and recognizing claw‐free graphs is still a challenge, where the current best deterministic algorithm for a graph G consists of performing executions of the best algorithm for matrix multiplication, we present linear‐time algorithms for CFVD on weighted block graphs and weighted graphs with bounded treewidth. Furthermore, we show that this problem on forests can be solved in linear time by a simpler algorithm, and we determine the exact values for full k‐ary trees. On the other hand, we show that CFVD is ‐hard even when the input graph is a split graph. We also show that the problem is hard to be approximated within any constant factor better than 2, assuming the unique games conjecture.

Equivalent polyadic decompositions of matrix multiplication tensors

Invariance transformations of polyadic decompositions of matrix multiplication tensors define an equivalence relation on the set of such decompositions. In this paper, we present an algorithm to efficiently decide whether two polyadic decompositions of a given matrix multiplication tensor are equivalent. With this algorithm, we analyze the equivalence classes of decompositions of several matrix multiplication tensors. This analysis is relevant for the study of fast matrix multiplication as it relates to the question of how many essentially different fast matrix multiplication algorithms there exist. This question has been first studied by de Groote, who showed that for the multiplication of 2 ×2 matrices with 7 active multiplications, all algorithms are essentially equivalent to Strassen’s algorithm. In contrast, the results of our analysis show that for the multiplication of larger matrices (e.g., 2 ×3 by 3 ×2 or 3 ×3 by 3 ×3 matrices), two decompositions are very likely to be essentially different. We further provide a necessary criterion for a polyadic decomposition to be equivalent to a polyadic decomposition with integer entries. Decompositions with specific integer entries, e.g., powers of two, provide fast matrix multiplication algorithms with better efficiency and stability properties. This condition can be tested algorithmically and we present the conclusions obtained for the decompositions of small/medium matrix multiplication tensors.

Universal points in the asymptotic spectrum of tensors

Motivated by the problem of constructing fast matrix multiplication algorithms, Strassen (FOCS 1986, Crelle 1987–1991) introduced and developed the theory of asymptotic spectra of tensors. For any sub-semiringX\mathcal {X}of tensors (under direct sum and tensor product), the duality theorem that is at the core of this theory characterizes basic asymptotic properties of the elements ofX\mathcal {X}in terms ofthe asymptotic spectrum ofX\mathcal {X}, which is defined as the collection of semiring homomorphisms fromX\mathcal {X}to the non-negative reals with a natural monotonicity property. The asymptotic properties characterized by this duality encompass fundamental problems in complexity theory, combinatorics and quantum information.Universal spectral pointsare elements in the asymptotic spectrum of the semiring ofalltensors. Finding all universal spectral points suffices to find the asymptotic spectrum ofanysub-semiring. The construction of non-trivial universal spectral points has been an open problem for more than thirty years. We construct, for the first time, a family of non-trivial universal spectral points over the complex numbers, calledquantum functionals. We moreover prove that the quantum functionals precisely characterise the asymptotic slice rank of complex tensors. Our construction, which relies on techniques from quantum information theory and representation theory, connects the asymptotic spectrum of tensors to the quantum marginal problem and entanglement polytopes.

A Mathematical Model for Determining the Composition of Binary Relations and an Algorithm for Binary Matrices Multiplication

Purpose of research. The need for the development of a mathematical model for determining the composition of binary relations and a hardware-oriented algorithm for multiplying binary matrices that allow organizing parallel data processing when determining the composition of binary relations.Methods. The procedure for determining the composition of binary relations is as follows: at the first stage, the definition of the sequence relation is performed, for this purpose its transitive closure is performed; the definition of the connection relation occurs after clarifying the sequence relation; then the alternative relation is clarified; at the final stage of determining the composition of binary relations, the parallelism relation is clarified.Results. In this paper, a mathematical model for determining the composition of binary relations and an algorithm for multiplying binary matrices are developed. The novelty of the mathematical model is the introduction of binary relations of connection, alternative and parallelism of the vertices of flowgraphs of parallel algorithms in addition to the sequence relation known in the graph theory. The novelty of the binary matrix multiplication algorithm is the reduction of the number of iterations of the inner cycle (with respect to the variable k) when obtaining a single value at one of the iterations of determining the scalar product of binary vectors.Conclusion. The developed mathematical model of binary relations of the sequence and connection of the vertices of flowgraphs of parallel algorithms allows for the organization of parallel data processing when determining the composition of binary relations. Based on the mathematical model for determining the composition of binary relations, a hardware-oriented algorithm for multiplying binary matrices was developed which allows transferring computationally complex procedures for multiplying binary matrices to the hardware level. The developed mathematical model and algorithm allow for the practical implementation of devices for multiplying binary matrices with an interruption of the internal cycle.

Dense Matrix Multiplication Algorithms and Performance Evaluation of HPCC in 81 Nodes IBM Power 8 Architecture

Optimizing HPC systems based on performance factors and bottlenecks is essential for designing an HPC infrastructure with the best characteristics and at a reasonable cost. Such insight can only be achieved through a detailed analysis of existing HPC systems and the execution of their workloads. The “Quinde I” is the only and most powerful supercomputer in Ecuador and is currently listed third on the South America. It was built with the IBM Power 8 servers. In this work, we measured its performance using different parameters from High-Performance Computing (HPC) to compare it with theoretical values and values obtained from tests on similar models. To measure its performance, we compiled and ran different benchmarks with the specific optimization flags for Power 8 to get the maximum performance with the current configuration in the hardware installed by the vendor. The inputs of the benchmarks were varied to analyze their impact on the system performance. In addition, we compile and compare the performance of two algorithms for dense matrix multiplication SRUMMA and DGEMM.

Extended use of error-free transformation for real matrix multiplication to complex matrix multiplication

The presence of rounding errors is frequently inevitable when performing arithmetic operations in computers due to the use of floating-point number system whose elements have finite precision. Consequently, error-free transformation algorithms are proposed as solutions, and a particular instance of these is the error-free transformation algorithm for real matrix multiplication. Based on the previously mentioned algorithm, this study proposes two error-error free transformation algorithms for complex matrix multiplication. By conducting the numerical experiment, one finds that the combination of both proposed algorithms with an accurate summation algorithm generates the most accurate results, but it also suffers from a notable increase in the computing time ratio. On the other hand, the second proposed algorithm, which is simply another form of the first one with the change of order in summation, combined with the pure floating-point addition produces results that are nearly as accurate as those generated by the former combination. Since these results are achieved with significantly smaller computing time ratio, then the study concludes that the last combination is the best method for complex matrix multiplication in terms of accuracy and computing time efficiency.

A Parallel Structured Divide-and-Conquer Algorithm for Symmetric Tridiagonal Eigenvalue Problems

In this paper, a parallel structured divide-and-conquer (PSDC) eigensolver is proposed for symmetric tridiagonal matrices based on ScaLAPACK and a parallel structured matrix multiplication algorithm, called PSMMA. Computing the eigenvectors via matrix-matrix multiplications is the most computationally expensive part of the divide-and-conquer algorithm, and one of the matrices involved in such multiplications is a rank-structured Cauchy-like matrix. By exploiting this particular property, PSMMA constructs the local matrices by using generators of Cauchy-like matrices without any communication, and further reduces the computation costs by using a structured low-rank approximation algorithm. Thus, both the communication and computation costs are reduced. Experimental results show that both PSMMA and PSDC are highly scalable and scale to 4096 processes at least. PSDC has better scalability than PHDC that was proposed in [J. Comput. Appl. Math. 344 (2018) 512--520] and only scaled to 300 processes for the same matrices. Comparing with \texttt{PDSTEDC} in ScaLAPACK, PSDC is always faster and achieves $1.4$x--$1.6$x speedup for some matrices with few deflations. PSDC is also comparable with ELPA, with PSDC being faster than ELPA when using few processes and a little slower when using many processes.

GCSA Codes With Noise Alignment for Secure Coded Multi-Party Batch Matrix Multiplication

A secure multi-party batch matrix multiplication problem (SMBMM) is considered, where the goal is to allow a master to efficiently compute the pairwise products of two batches of massive matrices, by distributing the computation across S servers. Any X colluding servers gain no information about the input, and the master gains no additional information about the input beyond the product. A solution called Generalized Cross Subspace Alignment codes with Noise Alignment (GCSA- NA) is proposed in this work, based on cross-subspace alignment codes. The state of art solution to SMBMM is a coding scheme called polynomial sharing (PS) that was proposed by Nodehi and Maddah-Ali. GCSA-NA outperforms PS codes in several key aspects—more efficient and secure inter-server communication, lower latency, flexible inter-server network topology, efficient batch processing, and tolerance to stragglers. The idea of noise alignment can also be combined with N-source Cross Subspace Alignment (N-CSA) codes and fast matrix multiplication algorithms like Strassen’s construction. Moreover, noise alignment can be applied to symmetric secure private information retrieval to achieve the asymptotic capacity.

Multi-Disease Classification Model Using Strassen’s Half of Threshold (SHoT) Training Algorithm in Healthcare Sector

In healthcare industry, Neural Network has attained a milestone in solving many real-life classification problems varies from very simple to complex and from linear to non-linear. To improve the training process by reducing the training time, Adaptive Skipping Training algorithm named as Half of Threshold (HOT) has been proposed. To perform the fast classification and also to improve the computational efficiency such as accuracy, error rate, etc., the highlighted characteristics of proposed HOT algorithm has been integrated with Strassen’s matrix multiplication algorithm and derived a novel, hybrid and computationally efficient algorithm for training and validating the neural network named as Strassen’s Half of Threshold (SHoT) Training Algorithm. The experimental outcome based on the simulation demonstrated that the proposed SHOT algorithm outperforms both BPN and HOT algorithm in terms of training time which is reduced with the range of 7% to 54% and its efficiency which is improved with the range of 3% to 15% on various dataset such as Hepatitis, SPeCT, Heart, Liver Disorders, Breast Cancer Wisconsin (Diagnostic), Drug Consumption, Cardiotocography, Splice-junction Gene Sequences and Thyroid Disease dataset that are extracted from Machine Learning Dataset Repository of UCI. It can be integrated with any type of supervised training algorithm.