Linear Algebra Computations Research Articles

Novel progressive deep learning algorithm for uncovering multiple single nucleotide polymorphism interactions to predict paclitaxel clearance in patients with nonsmall cell lung cancer.

The rate at which the anticancer drug paclitaxel is cleared from the body markedly impacts its dosage and chemotherapy effectiveness. Importantly, paclitaxel clearance varies among individuals, primarily because of genetic polymorphisms. This metabolic variability arises from a nonlinear process that is influenced by multiple single nucleotide polymorphisms (SNPs). Conventional bioinformatics methods struggle to accurately analyze this complex process and, currently, there is no established efficient algorithm for investigating SNP interactions. We developed a novel machine-learning approach called GEP-CSIs data mining algorithm. This algorithm, an advanced version of GEP, uses linear algebra computations to handle discrete variables. The GEP-CSI algorithm calculates a fitness function score based on paclitaxel clearance data and genetic polymorphisms in patients with nonsmall cell lung cancer. The data were divided into a primary set and a validation set for the analysis. We identified and validated 1184 three-SNP combinations that had the highest fitness function values. Notably, SERPINA1, ATF3 and EGF were found to indirectly influence paclitaxel clearance by coordinating the activity of genes previously reported to be significant in paclitaxel clearance. Particularly intriguing was the discovery of a combination of three SNPs in genes FLT1, EGF and MUC16. These SNPs-related proteins were confirmed to interact with each other in the protein-protein interaction network, which formed the basis for further exploration of their functional roles and mechanisms. We successfully developed an effective deep-learning algorithm tailored for the nuanced mining of SNP interactions, leveraging data on paclitaxel clearance and individual genetic polymorphisms.

A High-Speed Floating Point Matrix Multiplier Implemented in Reconfigurable Architecture

Matrix multiplication is a fundamental operation in computational applications across various domains. This paper introduces a novel reconfigurable co-processor that enhances the efficiency of matrix multiplication by concurrently executing addition and multiplication operations upon matrix elements of different sizes. The proposed design aims to reduce computation time and improve efficiency for matrix multiplication equations. Experimental evaluations were conducted on matrices of different sizes to demonstrate the effectiveness of the processor. The results reveal substantial improvements in both time and efficiency when compared to traditional approaches. The reconfigurable transformation processor harnesses parallel processing capabilities, enabling the simultaneous execution of addition and multiplication operations by partitioning input matrices into smaller submatrices and performing parallel computations, thus the processor achieves faster results. Additionally, the design incorporates configurable arithmetic units that dynamically adapt to matrix characteristics, further optimizing performance. The experimental evaluations provide evidence of reduction in computation time and improvement in efficiency. present significant benefits over traditional sequential methods. This makes this co-processor ideally fit for domains that require intensive linear algebra computations such as computer vision, machine learning, and signal processing.

Accurate Computations with Block Checkerboard Pattern Matrices

In this work, block checkerboard sign pattern matrices are introduced and analyzed. They satisfy the generalized Perron–Frobenius theorem. We study the case related to total positive matrices in order to guarantee bidiagonal decompositions and some linear algebra computations with high relative accuracy. A result on intervals of checkerboard matrices is included. Some numerical examples illustrate the theoretical results.

Block-wise dynamic mixed-precision for sparse matrix-vector multiplication on GPUs

Sparse matrix-vector multiplication (SpMV) plays a critical role in a wide range of linear algebra computations, particularly in scientific and engineering disciplines. However, the irregular memory access patterns, extensive memory usage, high bandwidth requirements, and underutilization of parallelism hinder the computational efficiency of SpMV on GPUs. In this paper, we propose a novel approach called block-wise dynamic mixed-precision (BDMP) to address these challenges. Our methodology involves partitioning the original matrix into uniformly sized blocks, with each block’s size determined by considering architectural characteristics and accuracy requirements. Additionally, we dynamically assign precision to each block using a precision selection method that takes into account the value distribution of the original sparse matrix. We develop two distinct SpMV computation algorithms for BDMP: BDMP-PBP (Precision-based partitioning) and BDMP-TCKI (Tailored compression and kernel implementation). BDMP-PBP partitions the matrix into two independent matrices for separate computations based on block precision, offering flexibility for integration with other optimization techniques. Meanwhile, BDMP-TCKI focuses on achieving significant thread-level parallelism and memory utilization by tailoring an appropriate compressed storage format and kernel implementation for each block. We compare BDMP with NVIDIA’s cuSPARSE library and three state-of-the-art SpMV methods, including SELLP, MergeBase, and BalanceCSR, using matrices from the University of Florida’s SuiteSparse dataset collection. BDMP-PBP and BDMP-TCKI show average speedups up to 2.64×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ imes $$\\end{document} and 2.91×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ imes $$\\end{document} on Turing RTX 2080Ti, and up to 2.99×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ imes $$\\end{document} and 3.22×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ imes $$\\end{document} on Ampere A100. The results demonstrate that BDMP enables the optimization of computation speed without compromising the precision necessary for reliable results.

Givens rotations for QR decomposition, SVD and PCA over database joins

This article introduces FiGaRo, an algorithm for computing the upper-triangular matrix in the QR decomposition of the matrix defined by the natural join over relational data. FiGaRo ’s main novelty is that it pushes the QR decomposition past the join. This leads to several desirable properties. For acyclic joins, it takes time linear in the database size and independent of the join size. Its execution is equivalent to the application of a sequence of Givens rotations proportional to the join size. Its number of rounding errors relative to the classical QR decomposition algorithms is on par with the database size relative to the join output size. The QR decomposition lies at the core of many linear algebra computations including the singular value decomposition (SVD) and the principal component analysis (PCA). We show how FiGaRo can be used to compute the orthogonal matrix in the QR decomposition, the SVD and the PCA of the join output without the need to materialize the join output. A suite of experiments validate that FiGaRo can outperform both in runtime performance and numerical accuracy the LAPACK library Intel MKL by a factor proportional to the gap between the sizes of the join output and input.

Алгоритмы дискретной фильтрации на основе модифицированной взвешенной ортогонализации Грама –Шмидта для дискретных стохастических систем с мультипликативными и аддитивными шумами

Discrete-time stochastic systems with multiplicative and additive noises describe a wide class of mathematical models of complex systems, for example, industrial-technological, energy, economic, telecommunication, aerospace systems, etc. An important class of algorithms for processing measure ment information in complex systems are Kalman-type discrete-time filtering algorithms. Purpose. Construction of new discrete-time filtering algorithms for discrete-time linear systems with multiplicative and additive noises based on numerically stable modified weighted Gram – Schmidt orthogonalization (MWGS). Methodology. The methods of computational linear algebra were used, namely, the direct procedu re of MWGS-orthogonalization, the theory of Kalman filtering, methods of scientific programming in MATLAB. Findings. New LD-algorithms for discrete-time filtering in covariance and informational form for discrete-time stochastic systems with multiplicative and additive noises are constructed. The algorithms have an extended array form allowing updates for all necessary filter values. The method employs numerically stable modified weighted Gram – Schmidt orthogonalization. Algebraic equiva lence of LD-filters to Kalman-type covariance and informational algorithms for linear discrete time stochastic systems with multiplicative and additive noises is proved. The conducted numerical experiments have shown the effectiveness of the proposed algorithms using the example of solving the problem of parametric estimation of a model of almost rectilinear motion, as well as their savings in computation time compared to previously constructed UD filters. Value. New discrete-time filtering LD-algorithms can be used as a reliable computational alterna tive to the Kalman-type “standard algorithms” since they have the property being numerically stable to machine round-off errors due to the use of the MWGS orthogonalization computational procedure at each iteration of the algorithm. The results can be used to solve problems of measurement information processing in discrete-time systems with multiplicative and additive noise.

Addition and intersection of linear time-invariant behaviors

We define and analyze the operations of addition and intersection of linear time-invariant systems in the behavioral setting, where systems are viewed as sets of trajectories rather than input–output maps. The classical definition of addition of input–output systems is addition of the outputs with the inputs being equal. In the behavioral setting, addition of systems is defined as addition of all variables. Intersection of linear time-invariant systems was considered before only for the autonomous case in the context of “common dynamics” estimation. We generalize the notion of common dynamics to open systems (systems with inputs) as intersection of behaviors. This is done by proposing trajectory-based definitions. The main results of the paper are (1) characterization of the link between the complexities (number of inputs and order) of the sum and intersection systems, (2) algorithms for computing their kernel and image representations and (3) a duality property of the two operations. Our approach combines polynomial and numerical linear algebra computations.

Improvement of variables interpretability in kernel PCA

BackgroundKernel methods have been proven to be a powerful tool for the integration and analysis of high-throughput technologies generated data. Kernels offer a nonlinear version of any linear algorithm solely based on dot products. The kernelized version of principal component analysis is a valid nonlinear alternative to tackle the nonlinearity of biological sample spaces. This paper proposes a novel methodology to obtain a data-driven feature importance based on the kernel PCA representation of the data.ResultsThe proposed method, kernel PCA Interpretable Gradient (KPCA-IG), provides a data-driven feature importance that is computationally fast and based solely on linear algebra calculations. It has been compared with existing methods on three benchmark datasets. The accuracy obtained using KPCA-IG selected features is equal to or greater than the other methods’ average. Also, the computational complexity required demonstrates the high efficiency of the method. An exhaustive literature search has been conducted on the selected genes from a publicly available Hepatocellular carcinoma dataset to validate the retained features from a biological point of view. The results once again remark on the appropriateness of the computed ranking.ConclusionsThe black-box nature of kernel PCA needs new methods to interpret the original features. Our proposed methodology KPCA-IG proved to be a valid alternative to select influential variables in high-dimensional high-throughput datasets, potentially unravelling new biological and medical biomarkers.

Task-based Parallel Programming for Scalable Matrix Product Algorithms

Task-based programming models have succeeded in gaining the interest of the high-performance mathematical software community because they relieve part of the burden of developing and implementing distributed-memory parallel algorithms in an efficient and portable way.In increasingly larger, more heterogeneous clusters of computers, these models appear as a way to maintain and enhance more complex algorithms. However, task-based programming models lack the flexibility and the features that are necessary to express in an elegant and compact way scalable algorithms that rely on advanced communication patterns. We show that the Sequential Task Flow paradigm can be extended to write compact yet efficient and scalable routines for linear algebra computations. Although, this work focuses on dense General Matrix Multiplication, the proposed features enable the implementation of more complex algorithms. We describe the implementation of these features and of the resulting GEMM operation. Finally, we present an experimental analysis on two homogeneous supercomputers showing that our approach is competitive up to 32,768 CPU cores with state-of-the-art libraries and may outperform them for some problem dimensions. Although our code can use GPUs straightforwardly, we do not deal with this case because it implies other issues which are out of the scope of this work.

Authenticated key agreement scheme for IoT networks exploiting lightweight linear algebraic computations

Authenticated key agreement scheme for IoT networks exploiting lightweight linear algebraic computations

Spectral Ranking in Complex Networks Using Memristor Crossbars

Various centrality measures have been proposed to identify the influence of each node in a complex network. Among the most popular ranking metrics, spectral measures stand out from the crowd. They rely on the computation of the dominant eigenvector of suitable matrices related to the graph: EigenCentrality, PageRank, Hyperlink Induced Topic Search (HITS) and Stochastic Approach for Link-Structure Analysis (SALSA). The simplest algorithm used to solve this linear algebra computation is the Power Method. It consists of multiple Matrix-Vector Multiplications (MVMs) and a normalization step to avoid divergent behaviours. In this work, we present an analog circuit used to accelerate the Power Iteration algorithm including current-mode termination for the memristor crossbars and a normalization circuit. The normalization step together with the feedback loop of the complete circuit ensure stability and convergence of the dominant eigenvector. We implement a transistor level peripheral circuitry around the memristor crossbar and take non-idealities such as wire parasitics, source driver resistance and finite memristor precision into account. We compute the different spectral centralities to demonstrate the performance of the system. We compare our results to the ones coming from the conventional digital computers and observe significant energy savings while maintaining a competitive accuracy.

A GPU parallel randomized CUR compression method for the Method of Moments

In this work, we propose a GPU parallel implementation of the randomized CUR (or Pseudo Skeleton) Approximation to compress the H-matrices of linear systems that arise in the discretization of integral equations modeling electromagnetic scattering problems. This compression method is highly parallelizable, in contrast with other similar methods such as the Adaptive Cross Approximation. It involves dense linear algebra computations that can be efficiently implemented on a GPU device. Besides, a stochastic convergence criterion is introduced to minimize the communication between the host and the device. Testing the code with standard cases shows the efficiency and accuracy of the method.

VCSR: An Efficient GPU Memory-Aware Sparse Format

The Sparse Matrix-Vector Multiplication (SpMV) kernel is used in a broad class of linear algebra computations. SpMV computations result in a performance bottleneck in many high performance applications, so optimizing SpMV performance is paramount. While implementing this kernel on a GPU can potentially boost performance significantly, current GPU libraries either provide modest performance gains or are burdened with high sparse format conversion overhead. In this paper we introduce the Vertical Compressed Sparse Row (VCSR) format, a novel memory-aware format that out-performs previous proposed formats on a GPU. We first motivate the design of our baseline VCSR format and then step through a series of enhancements that further improve VCSR's memory efficiency (VCSR-MEM) and performance (VCSR-INTRLV), while also considering conversion overhead. VCSR attempts to produce a high degree of thread-level parallelism and memory utilization by exploiting knowledge of GPU memory microarchitecture. VCSR can reduce the number of global memory transactions significantly, an issue not addressed by most other sparse formats. In addition, VCSR provides a novel reordering mechanism. It minimizes the size of the compressed matrix, handles both regular/irregular sparse matrices, and can be customized based on matrix size. VCSR also minimizes conversion overhead, as compared to full or partial row reordering. Our methodology is highly configurable and can be optimized for any sparse matrix. We have evaluated the VCSR format for the SpMV kernel when run on two different NVIDIA GPUs, the Kepler K40 and the Volta V100. We compare VCSR with NVIDIA's cuSPARSE library (the HYB format), a state-of-the-art sparse library. We also compare against other state-of-the-art CSR-based formats, including CSR5, merge-base SpMV and HOLA. We evaluate the benefits of VCSR over the entire University of Florida's SuiteSparse dataset collection. The VCSR-baseline format achieves an average speedup ranging from <inline-formula><tex-math notation="LaTeX">$1.10\times$</tex-math></inline-formula> to <inline-formula><tex-math notation="LaTeX">$1.39\times$</tex-math></inline-formula> when compared to the performance of the four state-of-the-art formats on an NVIDIA V100. While the VCSR-MEM format can save a significant amount of memory space, it is a bit slower than our VCSR-baseline. VCSR-INTRLV performs much better than the VCSR-baseline, and even when including the conversion overhead, achieves an average speedup of <inline-formula><tex-math notation="LaTeX">$1.08\times$</tex-math></inline-formula> as compared to HOLA (the best performing format among the prior schemes).

Energy Efficient Approximate 3D Image Reconstruction

We demonstrate an efficient and accelerated parallel, sparse depth reconstruction framework using compressed sensing (compressed sensing (CS)) and approximate computing. Employing data parallelism for rapid image formation, the depth image is reconstructed from sparsely sampled scenes using convex optimization. Coupled with faster imaging, this sparse sampling reduces significantly the projected laser power in active systems such as light detection and ranging (LiDAR) to allow eye safe operation at longer range. We also demonstrate how reduced precision is leveraged to reduce the number of logic units in field-programmable gate array (FPGA) implementations for such sparse imaging systems. It enables significant reduction in logic units, memory requirements and power consumption by over 80% with minimal impact on the quality of reconstruction. To further accelerate processing, pre-computed, important components of the lower-upper (LU) decomposition and other linear algebraic computations are used to solve the convex optimization problems. Our methodology is demonstrated by the application of the alternating direction method of multipliers (ADMM) and proximal gradient descent (PGD) algorithms. For comparison, a fully discrete least square reconstruction method ( <inline-formula><tex-math notation="LaTeX">$d$</tex-math></inline-formula> Sparse) is also presented. This demonstrates the feasibility of novel, high resolution, low power and high frame rate LiDAR depth imagers based on sparse illumination for use in applications where resources are strictly limited.

The Linear Algebra Mapping Problem. Current State of Linear Algebra Languages and Libraries

We observe a disconnect between developers and end-users of linear algebra libraries. On the one hand, developers invest significant effort in creating sophisticated numerical kernels. On the other hand, end-users are progressively less likely to go through the time consuming process of directly using said kernels; instead, languages and libraries, which offer a higher level of abstraction, are becoming increasingly popular. These languages offer mechanisms that internally map the input program to lower level kernels. Unfortunately, our experience suggests that, in terms of performance, this translation is typically suboptimal. In this paper, we define the problem of mapping a linear algebra expression to a set of available building blocks as the “Linear Algebra Mapping Problem” (LAMP) ; we discuss its NP-complete nature, and investigate how effectively a benchmark of test problems is solved by popular high-level programming languages and libraries. Specifically, we consider Matlab, Octave, Julia, R, Armadillo (C++), Eigen (C++), and NumPy (Python); the benchmark is meant to test both compiler optimizations, as well as linear algebra specific optimizations, such as the optimal parenthesization of matrix products. The aim of this study is to facilitate the development of languages and libraries that support linear algebra computations.

A survey on machine learning in array databases

This paper provides an in-depth survey on the integration of machine learning and array databases. First,machine learning support in modern database management systems is introduced. From straightforward implementations of linear algebra operations in SQL to machine learning capabilities of specialized database managers designed to process specific types of data, a number of different approaches are overviewed. Then, the paper covers the database features already implemented in current machine learning systems. Features such as rewriting, compression, and caching allow users to implement more efficient machine learning applications. The underlying linear algebra computations in some of the most used machine learning algorithms are studied in order to determine which linear algebra operations should be efficiently implemented by array databases. An exhaustive overview of array data and relevant array database managers is also provided. Those database features that have been proven of special importance for efficient execution of machine learning algorithms are analyzed in detail for each relevant array database management system. Finally, current state of array databases capabilities for machine learning implementation is shown through two example implementations in Rasdaman and SciDB.

A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level

We demonstrate that a neural network pretrained on text and fine-tuned on code solves mathematics course problems, explains solutions, and generates questions at a human level. We automatically synthesize programs using few-shot learning and OpenAI's Codex transformer and execute them to solve course problems at 81% automatic accuracy. We curate a dataset of questions from Massachusetts Institute of Technology (MIT)'s largest mathematics courses (Single Variable and Multivariable Calculus, Differential Equations, Introduction to Probability and Statistics, Linear Algebra, and Mathematics for Computer Science) and Columbia University's Computational Linear Algebra. We solve questions from a MATH dataset (on Prealgebra, Algebra, Counting and Probability, Intermediate Algebra, Number Theory, and Precalculus), the latest benchmark of advanced mathematics problems designed to assess mathematical reasoning. We randomly sample questions and generate solutions with multiple modalities, including numbers, equations, and plots. The latest GPT-3 language model pretrained on text automatically solves only 18.8% of these university questions using zero-shot learning and 30.8% using few-shot learning and the most recent chain of thought prompting. In contrast, program synthesis with few-shot learning using Codex fine-tuned on code generates programs that automatically solve 81% of these questions. Our approach improves the previous state-of-the-art automatic solution accuracy on the benchmark topics from 8.8 to 81.1%. We perform a survey to evaluate the quality and difficulty of generated questions. This work automatically solves university-level mathematics course questions at a human level and explains and generates university-level mathematics course questions at scale, a milestone for higher education.

Towards electronic structure-based ab-initio molecular dynamics simulations with hundreds of millions of atoms

We push the boundaries of electronic structure-based ab-initio molecular dynamics (AIMD) beyond 100 million atoms. This scale is otherwise barely reachable with classical force-field methods or novel neural network and machine learning potentials. We achieve this breakthrough by combining innovations in linear-scaling AIMD, efficient and approximate sparse linear algebra, low and mixed-precision floating-point computation on GPUs, and a compensation scheme for the errors introduced by numerical approximations. The core of our work is the non-orthogonalized local submatrix method (NOLSM), which scales very favorably to massively parallel computing systems and translates large sparse matrix operations into highly parallel, dense matrix operations that are ideally suited to hardware accelerators. We demonstrate that the NOLSM method, which is at the center point of each AIMD step, is able to achieve a sustained performance of 324 PFLOP/s in mixed FP16/FP32 precision corresponding to an efficiency of 67.7% when running on 1536 NVIDIA A100 GPUs.

Accelerating approximate matrix multiplication for near-sparse matrices on GPUs

Although the matrix multiplication plays a vital role in computational linear algebra, there are few efficient solutions for matrix multiplication of the near-sparse matrices. The Sparse Approximate Matrix Multiply (SpAMM) is one of the algorithms to fill the performance gap neglected by traditional optimizations for dense/sparse matrix multiplication. However, existing SpAMM algorithms fail to exploit the performance potential of GPUs for acceleration. In this paper, we present cuSpAMM, the first parallel SpAMM algorithm optimized for multiple GPUs. Several performance optimizations have been proposed, including algorithm re-design to adapt to the thread parallelism, blocking strategies for memory access optimization, and the acceleration with the tensor core. In addition, we scale cuSpAMM to run on multiple GPUs with an effective load balance scheme. We evaluate cuSpAMM on both synthesized and real-world datasets on multiple GPUs. The experiment results show that cuSpAMM achieves significant performance speedup compared to vendor optimized cuBLAS and cuSPARSE libraries.

A Parallel Approach of the Enhanced Craig–Bampton Method

The enhanced Craig–Bampton (ECB) method is a novel extension of the original Craig–Bampton (CB) method, which has been widely used for component mode synthesis (CMS). The ECB method, using residual modal compensation that is neglected in the CB method, provides dramatic accuracy improvement of reduced matrices without an increasing number of eigenbasis. However, it also needs additional computational requirements to treat the residual flexibility. In this paper, an efficient parallelization of the ECB method is presented to handle this issue and accelerate the applicability for large-scale structural vibration problems. A new ECB formulation within a substructuring strategy is derived to achieve better scalability. The parallel implementation is based on OpenMP parallel architecture. METIS graph partitioning and Linear Algebra Package (LAPACK) are used to automated algebraic partitioning and computational linear algebra, respectively. Numerical examples are presented to evaluate the accuracy, scalability, and capability of the proposed parallel ECB method. Consequently, based on this work, one can expect effective computation of the ECB method as well as accuracy improvement.