Strassen's Matrix Research Articles

Minimizing Multiplication of Kernel Computation in Convolutional Neural Networks Using Strassen Algorithm

Convolution neural networks (CNN) have been widely applied for the computer vision task. However, the success of CNN is limited by the computational complexity of the network, so it is difficult for the model to run the inference process in real time. In this paper, we apply Strassen matrix multiplication to reduce multiplications in convolution operations in CNN, in order to get faster execution for CNN. First, we transform the convolution operation into a matrix multiplication operation using the Toeplitz mapping method, then after that, we apply the Strassen method to these matrices. In the end, we compare the number of arithmetic operations (multiplication and addition) in the convolutional layer using Strassen and the standard algorithm. We apply this algorithm implementation in convolution layers 1 and 3 in LeNet-5 Architecture.

Grothendieck constant is norm of Strassen matrix multiplication tensor

We show that two important quantities from two disparate areas of complexity theory --- Strassen's exponent of matrix multiplication $\omega$ and Grothendieck's constant $K_G$ --- are intimately related. They are different measures of size for the same underlying object --- the matrix multiplication tensor, i.e., the $3$-tensor or bilinear operator $\mu_{l,m,n} : \mathbb{F}^{l \times m} \times \mathbb{F}^{m \times n} \to \mathbb{F}^{l \times n}$, $(A,B) \mapsto AB$ defined by matrix-matrix product over $\mathbb{F} = \mathbb{R}$ or $\mathbb{C}$. It is well-known that Strassen's exponent of matrix multiplication is the greatest lower bound on (the log of) a tensor rank of $\mu_{l,m,n}$. We will show that Grothendieck's constant is the least upper bound on a tensor norm of $\mu_{l,m,n}$, taken over all $l, m, n \in \mathbb{N}$. Aside from relating the two celebrated quantities, this insight allows us to rewrite Grothendieck's inequality as a norm inequality \[ \lVert\mu_{l,m,n}\rVert_{1,2,\infty} =\max_{X,Y,M\neq0}\frac{|\operatorname{tr}(XMY)|}{\lVert X\rVert_{1,2}\lVert Y\rVert_{2,\infty}\lVert M\rVert_{\infty,1}}\le K_G. \] We prove that Grothendieck's inequality is unique: If we generalize the $(1,2,\infty)$-norm to arbitrary $p,q, r \in [1, \infty]$, \[ \lVert\mu_{l,m,n}\rVert_{p,q,r}=\max_{X,Y,M\neq0}\frac{|\operatorname{tr}(XMY)|}{\|X\|_{p,q}\|Y\|_{q,r}\|M\|_{r,p}}, \] then $(p,q,r )=(1,2,\infty)$ is, up to cyclic permutations, the only choice for which $\lVert\mu_{l,m,n}\rVert_{p,q,r}$ is uniformly bounded by a constant independent of $l,m,n$.

A data-driven paradigm for mapping problems

We propose data-driven paradigm for solving task mapping problems.Data-mapping paradigm is formulated for multiple-core systems.Data dependency analysis replaced task dependency analysis on mapping problems.We prototype data-mapping paradigm for Strassen and Cholesky inversion methods.Data-mapping paradigm is an efficient approach for data-intensive applications. We present a new data-driven paradigm for solving mapping problems on parallel computers. This paradigm targets at mapping data modules, instead of task modules, onto multiple processing cores. By dependency analysis of data modules, we devise a data movement matrix to reduce the need of manipulating task program modules at the expenses of handling data modules. To visualize and quantify the complex maneuver, we adopt the parallel activities trace graphs introduced earlier. To demonstrate the procedure and algorithmic values of our paradigm, we test it on the Strassen matrix multiplication and Cholesky matrix inversion algorithms. Mapping tasks has been more widely studied while mapping data is a new approach that appears to be more efficient for data-intensive applications that are becoming prevalent for today's parallel computers with millions of cores.

Determining the Effects of Cross-over Point on the Running Time of Strassen Matrix Multiplication Algorithm

This paper studies Strassen’s algorithms for fast multiplication of two finite dimensional matrices. However, one pertinent issue that has deterred Strassen’s scheme from been considered for practical usage is determining the cross-over point. In this light, large matrices with different sizes were randomly generated on which Strassen and conventional matrix multiplication algorithms were implemented in MATLAB R2008b. Two MATLAB built-in functions nextpow2 and pow2 were used for implementing padding techniques to ensure that the matrices are to the power of two. Three different experiments were carried out using five, four and three levels of recursion (divide and conquer algorithm) respectively to determine the suitable cut-off point which were used to evaluate the optimal running time for Strassen’s algorithm. For each experiment, eight finite dimensional square matrices of real numbers were generated and iteratively multiplied. The experiment reveals that the cut-off point with five level of recursion optimized the Strassens time.

A Cloud-Friendly Communication-Optimal Implementation for Strassen's Matrix Multiplication Algorithm

Due to its on-demand and pay-as-you-go properties, cloud computing has become an attractive alternative for HPC applications. However, communication-intensive applications with complex communication patterns still cannot be performed efficiently on cloud platforms, which are equipped with MapReduce technologies, such as Hadoop and Spark. In particular, one major obstacle is that MapReduce's simple programming model cannot explicitly manipulate data transfers between compute nodes. Another obstacle is cloud's relatively poor network performance compared with traditional HPC platforms. The traditional Strassen's algorithm of square matrix multiplication has a recursive and complex pattern on the HPC platform. Therefore, it cannot be directly applied to the cloud platform. In this paper, we demonstrate how to make Strassen's algorithm with complex communication patterns “cloud-friendly”. By reorganizing Strassen's algorithm in an iterative pattern, we completely separate its computations and communications, making it fit to MapReduce programming model. By adopting a novel data/task parallel strategy, we solve Strassen's data dependency problems, making it well balanced. This is the first instance of Strassen's algorithm in MapReduce-style systems, which also matches Strassen's communication lower bound. Further experimental results show that it achieves a speedup ranging from 1.03× to 2.50× over the classical Θ(n3) algorithm. We believe the principle can be applied to many other complex scientific applications.

Communication costs of Strassen's matrix multiplication

Algorithms have historically been evaluated in terms of the number of arithmetic operations they performed. This analysis is no longer sufficient for predicting running times on today's machines. Moving data through memory hierarchies and among processors requires much more time (and energy) than performing computations. Hardware trends suggest that the relative costs of this communication will only increase. Proving lower bounds on the communication of algorithms and finding algorithms that attain these bounds are therefore fundamental goals. We show that the communication cost of an algorithm is closely related to the graph expansion properties of its corresponding computation graph. Matrix multiplication is one of the most fundamental problems in scientific computing and in parallel computing. Applying expansion analysis to Strassen's and other fast matrix multiplication algorithms, we obtain the first lower bounds on their communication costs. These bounds show that the current sequential algorithms are optimal but that previous parallel algorithms communicate more than necessary. Our new parallelization of Strassen's algorithm is communication-optimal and outperforms all previous matrix multiplication algorithms.<!-- END_PAGE_1 -->

Block recursive computation of generalized inverses

C � Abstract. A fully block recursive method for computing outer generalized inverses of given square matrix is introduced. The method is applicable even in the case when some of main diagonal minors of A are singular or A is singular. Computational complexity of the method is not harder than the matrix multiplication, under the assumption that the Strassen matrix inversion algorithm is used. A partially recursive algorithm for computing various classes of generalized inverses is also developed. This method can be efficiently used for the acceleration of the known methods for computing generalized inverses.

An Efficient DCT Compression Technique using Strassen's Matrix Multiplication Algorithm

compression' minimizes the problem that we face in storing and transmitting large amount of data. It reduces the size of data required to represent a digital image. In this procedure, DCT plays an important role. It separates information by using different frequencies. In Discrete Cosine Transformation (DCT), Quantization and encoding are the steps involved in the compression of the JPEG image. In this whole work, while using DCT, we have used Stassen's matrix multiplication algorithm for reducing the complex matrix multiplication problems. As per the result obtained from experiment, the performance of DCT is improved by using Stassen's matrix multiplication algorithm. The performance analysis is carried out through Peak signal to noise ratio (PSNR), and the different compression ratio (CR) for the different images. Keywords— compression, strassen's matrix multiplication, CR, DCT, JPEG, PSNR.

A parallel implementation of Strassen’s matrix multiplication algorithm for wormhole-routed all-port 2D torus networks

A new parallel implementation of Strassen's matrix multiplication algorithm is proposed for massively parallel supercomputers with 2D, all-port torus interconnection networks. The proposed algorith...

A High Security Approach for Image Steganography by HIOP Algorithm and Strassen's Matrix Multiplication and Rapid Information Retrieval

There are many ways to hide information or transmission of information secretly. In this sense steganography is the best part of sending information secretly. The technology has certainly been the topic of widespread discussion among the IT community. Steganography, like watermarking and fingerprinting is a branch of science to hide information but, unlike watermarking and fingerprinting, steganogaraphy is the art ofwriting message or information in such a way that no one apart suitable recipient knows the meaning of the message or information. There are many techniques to performsteganography on electronic media, most notably audio and image files. The outcome of thispaper is to generate a cross platformthat can effectively hide a message inside a digitalimage file. We are presenting a technique which works by changing a few pixel color value at selected pixel in the image. We divide the image into blocks and in each block, the pixels are selected using HIOP (Higher Intensity of Pixel) algorithm, based on consideration the criteria of color intensity on image. To increase the security level, we create distribution in selected pixels whit using strassen multiplication in each block. Also, when retrieving a message for avoid repeating process which hidden text storage and speed up information retrieval, we save address of selected pixel. It is also, try not to degrade image quality and as far as possible does not change the image size. Key words: Steganography, watermarking, fingerprinting, HIOP, strassen multiplication, color intensity.

From program verification to program synthesis

This paper describes a novel technique for the synthesis of imperative programs. Automated program synthesis has the potential to make programming and the design of systems easier by allowing programs to be specified at a higher-level than executable code. In our approach, which we call proof-theoretic synthesis, the user provides an input-output functional specification, a description of the atomic operations in the programming language, and a specification of the synthesized program's looping structure, allowed stack space, and bound on usage of certain operations. Our technique synthesizes a program, if there exists one, that meets the input-output specification and uses only the given resources. The insight behind our approach is to interpret program synthesis as generalized program verification, which allows us to bring verification tools and techniques to program synthesis. Our synthesis algorithm works by creating a program with unknown statements, guards, inductive invariants, and ranking functions. It then generates constraints that relate the unknowns and enforces three kinds of requirements: partial correctness, loop termination, and well-formedness conditions on program guards. We formalize the requirements that program verification tools must meet to solve these constraint and use tools from prior work as our synthesizers. We demonstrate the feasibility of the proposed approach by synthesizing programs in three different domains: arithmetic, sorting, and dynamic programming. Using verification tools that we previously built in the VS3 project we are able to synthesize programs for complicated arithmetic algorithms including Strassen's matrix multiplication and Bresenham's line drawing; several sorting algorithms; and several dynamic programming algorithms. For these programs, the median time for synthesis is 14 seconds, and the ratio of synthesis to verification time ranges between 1x to 92x (with an median of 7x), illustrating the potential of the approach.

Comment on “Optimized matrix inversion technique for the T-matrix method”

The matrix inversion technique for the T-matrix method recently suggested by Petrov et al. [Opt. Lett.32, 1168 (2007)] is not new. Their way of implementing the technique, in which Strassen's matrix multiplication algorithm is not adopted, does not save computation time when the LU factorization technique of matrix computation theory is used.

Multi-Terminal Binary Decision Diagrams: An Efficient DataStructure for Matrix Representation

In this paper, we discuss the use of binary decision diagrams to represent general matrices. We demonstrate that binary decision diagrams are an efficient representation for every special-case matrix in common use, notably sparse matrices. In particular, we demonstrate that for any matrix, the BDD representation can be no larger than the corresponding sparse-matrix representation. Further, the BDD representation is often smaller than any other conventional special-case representation: for the n×n Walsh matrix, for example, the BDD representation is of size O(log n). No other special-case representation in common use represents this matrix in space less than O(n²). We describe termwise, row, column, block, and diagonal selection over these matrices, standard an Strassen matrix multiplication, and LU factorization. We demonstrate that the complexity of each of these operations over the BDD representation is no greater than that over any standard representation. Further, we demonstrate that complete pivoting is no more difficult over these matrices than partial pivoting. Finally, we consider an example, the Walsh Spectrum of a Boolean function.

A Framework for Generating Distributed-Memory Parallel Programs for Block Recursive Algorithms

A framework for synthesizing communication-efficient distributed-memory parallel programs for block recursive algorithms such as the fast Fourier transform (FFT) and Strassen's matrix multiplication is presented. This framework is based on an algebraic representation of the algorithms, which involves the tensor (Kronecker) product and other matrix operations. This representation is useful in analyzing the communication implications of computation partitioning and data distributions. The programs are synthesized under two different target program models. These two models are based on different ways of managing the distribution of data for optimizing communication. The first model uses point-to-point interprocessor communication primitives, whereas the second model uses data redistribution primitives involving collective all-to-many communication. These two program models are shown to be suitable for different ranges of problem size. The methodology is illustrated by synthesizing communication-efficient programs for the FFT. This framework has been incorporated into the EXTENT system for automatic generation of parallel/vector programs for block recursive algorithms.

Using Strassen's matrix multiplication in high performance solution of linear systems

Performance characteristics of dense and structured blocked linear system solvers are studied when Strassen's matrix multiplication is used in the update step. Results of experiments on a multiprocessor Cray Y-MP are presented and discussed.

A RECURRENCE-FREE VARIANT OF STRASSEN'S ALGORITHM ON HYPERCUBE∗

In this paper a non-recursive Strassen's matrix multiplication algorithm is presented. This new algorithm is suitable to run on parallel environments. Two computational schemes have been worked out exploiting different parallel approaches on hypercube architecture. A comparative analysis is reported. The experiments have been carried out on an nCUBE-2 supercomputer, housed at CNUCE in Pisa, Express parallel operating system.

Comparisons of Gaussian elimination algorithms on a cray Y-MP

The results of various implementations of Gaussian elimination on full matrices on a single processor Cray Y-MP are presented and discussed. It is shown that when the manufacturer supplied BLAS kernels are used, the difference between thebest versions of level 2 BLAS and level 3 BLAS (blocked) implementations is almost negligible. It is suggested that to improve the performance of blocked Gaussian elimination it is possible to utilize Strassen's matrix multiplication algorithm.

Multilinear algebra and parallel programming

We discuss a programming methodology based on the use of multilinear algebra to design and implement parallel algorithms for linear computations. In particular, we review techniques for implementing expressions involving the tensor product. We then show how the tensor product can be used to formulate Strassen's matrix multiplication algorithm. We report on our experience using this formulation and these techniques to implement a parallel version of Strassen's matrix multiplication algorithm on the Encore Multimax.

A report on the performance of an implementation of Strassen's algorithm

We report some experimental results on the performance of several sequential and parallel programs implementing the tensor product formulation of Strassen's matrix multiplication algorithm that we presented recently in this journal [1].

A tensor product formulation of Strassen's matrix multiplication algorithm

Tensor product notation is used to derive an iterative version of Strassen's matrix multiplication algorithm.