Multiplication Of Large Matrices Research Articles

Privacy preserving image registration

Image registration is a key task in medical imaging applications, allowing to represent medical images in a common spatial reference frame. Current approaches to image registration are generally based on the assumption that the content of the images is usually accessible in clear form, from which the spatial transformation is subsequently estimated. This common assumption may not be met in practical applications, since the sensitive nature of medical images may ultimately require their analysis under privacy constraints, preventing to openly share the image content. In this work, we formulate the problem of image registration under a privacy preserving regime, where images are assumed to be confidential and cannot be disclosed in clear. We derive our privacy preserving image registration framework by extending classical registration paradigms to account for advanced cryptographic tools, such as secure multi-party computation and homomorphic encryption, that enable the execution of operations without leaking the underlying data. To overcome the problem of performance and scalability of cryptographic tools in high dimensions, we propose several techniques to optimize the image registration operations by using gradient approximations, and by revisiting the use of homomorphic encryption trough packing, to allow the efficient encryption and multiplication of large matrices. We focus on registration methods of increasing complexity, including rigid, affine, and non-linear registration based on cubic splines or diffeomorphisms parameterized by time-varying velocity fields. In all these settings, we demonstrate how the registration problem can be naturally adapted for accounting to privacy-preserving operations, and illustrate the effectiveness of PPIR on a variety of registration tasks.

The Law of Multiplication of Large Random Matrices Revisited

The Law of Multiplication of Large Random Matrices Revisited

A Nonvolatile All-Spin Nonbinary Matrix Multiplier: An Efficient Hardware Accelerator for Machine Learning

We propose and analyze a compact and non-volatile nanomagnetic (all-spin) non-binary matrix multiplier performing the multiply-and-accumulate (MAC) operation using two magnetic tunnel junctions - one activated by strain to act as the multiplier, and the other activated by spin-orbit torque pulses to act as a domain wall synapse that performs the operation of the accumulator. It has two advantages over the usual crossbar-based electronic non-binary matrix multiplier. First, while the crossbar architecture requires N3 devices to multiply two matrices, we require only 2N2 devices. Second, our matrix multiplier is non-volatile and retains the information about the product matrix after being powered off. Here, we present an example where each MAC operation can be performed in ~5 ns and the maximum energy dissipated per operation is ~60Nmax aJ, where Nmax is the largest matrix size. This provides a very useful hardware accelerator for machine learning and artificial intelligence tasks which involve the multiplication of large matrices. The non-volatility allows the matrix multiplier to be embedded in powerful non-von-Neumann architectures, including processor-in-memory. It also allows much of the computing to be done at the edge (of internet-of-things) while reducing the need to access the cloud, thereby making artificial intelligence more resilient against cyberattacks.

Robust Unit-Level Small Area Estimation: A Fast Algorithm for Large Datasets

Small area estimation is a topic of increasing importance in official statistics. Although the classical EBLUP method is useful for estimating the small area means efficiently under the normality assumptions, it can be highly influenced by the presence of outliers. Therefore, Sinha and Rao (2009; The Canadian Journal of Statistics) proposed robust estimators/predictors for a large class of unit- and area-level models. We confine attention to the basic unit-level model and discuss a related, but slightly different, robustification. In particular, we develop a fast algorithm that avoids inversion and multiplication of large matrices, and thus permits the user to apply the method to large datasets. In addition, we derive much simpler expressions of the boundedinfluence predicting equations to robustly predict the small-area means than Sinha and Rao (2009) did.

Improving Matrix Multiplication Using Parallel Computing

Multiplication of large matrices requires a lot of computation time as its complexity is O(n3). Because most image processing applications require higher computational throughputs with minimum time, many sequential and parallel algorithms are developed. In this paper, a method of matrix multiplication was chosen, and analyzed. A performance analysis was evaluated, and it was seen that the chosen method was very powerful when dealing with matrices with large sizes and implementing the method using parallel computing based on openMP libraries.

A practical streaming approximate matrix multiplication algorithm

Approximate Matrix Multiplication (AMM) has emerged as a useful and computationally inexpensive substitute for actual multiplication of large matrices. Randomized as well as deterministic solutions to AMM were provided in the past. The latest work provides a deterministic algorithm that solves AMM more accurately than the other works. It is a streaming algorithm that is both fast and accurate. But, it is less robust to noise and is also liable to have less than optimal performance in the presence of concept drift in the input matrices. We propose an algorithm that is more accurate, robust to noise, invariant to concept drift in the data, while having almost the same running time as the state-of-the-art algorithm. We also prove that theoretical guarantees exist for the proposed algorithm. An empirical performance improvement of up to 90% is obtained over the previous algorithm. We also propose a general framework for parallelizing the proposed algorithm. The two parallelized versions of the algorithm achieve up to 1.9x and 3.6x speedups over the original version of the proposed algorithm.

Out-of-core implementation for accelerator kernels on heterogeneous clouds

Cloud environments today are increasingly featuring hybrid nodes containing multicore CPU processors and a diverse mix of accelerators such as Graphics Processing Units (GPUs), Intel Xeon Phi co-processors, and Field-Programmable Gate Arrays (FPGAs) to facilitate easier migration to them of HPC workloads. While virtualization of accelerators in clouds is a leading research challenge, we address the programming challenges that assail execution of large instances of data-parallel applications using these accelerators in this paper. In a typical hybrid node in a cloud, the tight integration of accelerators with multicore CPUs via PCI-E communication links contains inherent limitations such as limited main memory of accelerators and limited bandwidth of the PCI-E communication links. These limitations poses formidable programming challenges to execution of large problem sizes on these accelerators. In this paper, we describe a library containing interfaces (HCLOOC) that addresses these challenges. It employs optimal software pipelines to overlap data transfers between host CPU and the accelerator and computations on the accelerator. It is designed using the fundamental building blocks, which are OpenCL command queues for FPGAs, Intel offload streams for Intel Xeon Phis, and CUDA streams and events that allow concurrent utilization of the copy and execution engines provided in NVidia GPUs. We elucidate the key features of our library using an out-of-core implementation of matrix multiplication of large dense matrices on a hybrid node, an Intel Haswell multicore CPU server hosting three accelerators that includes NVidia K40c GPU, Intel Xeon Phi 3120P, and a Xilinx FPGA. Based on experiments with the GPU, we show that our out-of-core implementation achieves 82% of peak double-precision floating performance of the GPU and a speedup of 2.7 times over the NVidia’s out-of-core matrix multiplication implementation (CUBLAS-XT). We also demonstrate that our implementation exhibits 0% drop in performance when the problem size exceeds the main memory of the GPU. We observe this 0% drop also for our implementation for Intel Xeon Phi and Xilinx FPGA.

RSTensorFlow: GPU Enabled TensorFlow for Deep Learning on Commodity Android Devices.

Mobile devices have become an essential part of our daily lives. By virtue of both their increasing computing power and the recent progress made in AI, mobile devices evolved to act as intelligent assistants in many tasks rather than a mere way of making phone calls. However, popular and commonly used tools and frameworks for machine intelligence are still lacking the ability to make proper use of the available heterogeneous computing resources on mobile devices. In this paper, we study the benefits of utilizing the heterogeneous (CPU and GPU) computing resources available on commodity android devices while running deep learning models. We leveraged the heterogeneous computing framework RenderScript to accelerate the execution of deep learning models on commodity Android devices. Our system is implemented as an extension to the popular open-source framework TensorFlow. By integrating our acceleration framework tightly into TensorFlow, machine learning engineers can now easily make benefit of the heterogeneous computing resources on mobile devices without the need of any extra tools. We evaluate our system on different android phones models to study the trade-offs of running different neural network operations on the GPU. We also compare the performance of running different models architectures such as convolutional and recurrent neural networks on CPU only vs using heterogeneous computing resources. Our result shows that although GPUs on the phones are capable of offering substantial performance gain in matrix multiplication on mobile devices. Therefore, models that involve multiplication of large matrices can run much faster (approx. 3 times faster in our experiments) due to GPU support.

Efficient carrier frequency offset estimation for orthogonal frequency‐division multiple access uplink with an arbitrary number of subscriber stations

An efficient method is proposed to estimate the carrier frequency offsets (CFOs) in the orthogonal frequency-division multiple access (OFDMA) uplink. The conventional alternating projection method is accelerated by utilising the inherited properties of the matrices involved. The multiplication of large sparse projection matrices can be elegantly transformed to a series of products involving small dense matrices, and the inverse operation of these large matrices can be substituted by direct computations. Hence, the computational cost is significantly reduced without compromising the accuracy of the CFO estimation.

Commutative law for products of infinitely large isotropic random matrices

Ensembles of isotropic random matrices are defined by the invariance of the probability measure under the left (and right) multiplication by an arbitrary unitary matrix. We show that the multiplication of large isotropic random matrices is spectrally commutative and self-averaging in the limit of infinite matrix size N→∞. The notion of spectral commutativity means that the eigenvalue density of a product ABC... of such matrices is independent of the order of matrix multiplication, for example, the matrix ABCD has the same eigenvalue density as ADCB. In turn, the notion of self-averaging means that the product of n independent but identically distributed random matrices, which we symbolically denote by AAA..., has the same eigenvalue density as the corresponding power A(n) of a single matrix drawn from the underlying matrix ensemble. For example, the eigenvalue density of ABCCABC is the same as that of A(2)B(2)C(3). We also discuss the singular behavior of the eigenvalue and singular value densities of isotropic matrices and their products for small eigenvalues λ→0. We show that the singularities at the origin of the eigenvalue density and of the singular value density are in one-to-one correspondence in the limit N→∞: The eigenvalue density of an isotropic random matrix has a power-law singularity at the origin ~|λ|(-s) with a power sε(0,2) when and only when the density of its singular values has a power-law singularity ~λ(-σ) with a power σ=s/(4-s). These results are obtained analytically in the limit N→∞. We supplement these results with numerical simulations for large but finite N and discuss finite-size effects for the most common ensembles of isotropic random matrices.

Extending the length and time scales of Gram–Schmidt Lyapunov vector computations

Lyapunov vectors have found growing interest recently due to their ability to characterize systems out of thermodynamic equilibrium. The computation of orthogonal Gram–Schmidt vectors requires multiplication and QR decomposition of large matrices, which grow as N2 (with the particle count). This expense has limited such calculations to relatively small systems and short time scales. Here, we detail two implementations of an algorithm for computing Gram–Schmidt vectors. The first is a distributed-memory message-passing method using Scalapack. The second uses the newly-released MAGMA library for GPUs. We compare the performance of both codes for Lennard–Jones fluids from N=100 to 1300 between Intel Nahalem/Infiniband DDR and NVIDIA C2050 architectures. To our best knowledge, these are the largest systems for which the Gram–Schmidt Lyapunov vectors have been computed, and the first time their calculation has been GPU-accelerated. We conclude that Lyapunov vector calculations can be significantly extended in length and time by leveraging the power of GPU-accelerated linear algebra.

PARAFAC algorithms for large-scale problems

Parallel factor analysis (PARAFAC) is a tensor (multiway array) factorization method which allows to find hidden factors (component matrices) from a multidimensional data. Most of the existing algorithms for the PARAFAC, especially the alternating least squares (ALS) algorithm need to compute Khatri–Rao products of tall factors and multiplication of large matrices, and due to this require high computational cost and large memory and are not suitable for very large-scale-problems. Hence, PARAFAC for large-scale data tensors is still a challenging problem. In this paper, we propose a new approach based on a modified ALS algorithm which computes Hadamard products, instead Khatri–Rao products, and employs relatively small matrices. The new algorithms are able to process extremely large-scale tensors with billions of entries. Extensive experiments confirm the validity and high performance of the developed algorithm in comparison with other well-known algorithms.

Parallel matrix multiplication on networked microcomputers

We present methods for utilizing parallel processing capability of the idle microcomputers on a local area network to perform computationally intensive operations frequently encountered in linear algebra. By making use of the data communication properties of the network, parallel algorithms for multiplication of large matrices have been designed and implemented. The techniques presented in this paper can be used to obtain efficient implementations of basic linear algebra subroutines.

Partial and Total Matrix Multiplication

In 1979 considerable progress was made in estimating the complexity of matrix multiplication. Here the new techniques and recent results are presented, based upon the notion of approximate rank and the observation that certain patterns of partial matrix multiplication (some of the entries of the matrices may be zero) can efficiently be utilized to perform multiplication of large total matrices. By combining Pan’s trilinear technique with a strong version of our compression theorem for the case of several disjoint matrix multiplications it is shown that multiplication of $N \times N$ matrices (over arbitrary fields) is possible in time $O(N^\beta )$, where $\beta $ is a bit smaller than $3\ln 52/\ln 110 \approx 2.522$.