BLAS Research Articles

Fast and exact fixed-radius neighbor search based on sorting.

Fixed-radius near neighbor search is a fundamental data operation that retrieves all data points within a user-specified distance to a query point. There are efficient algorithms that can provide fast approximate query responses, but they often have a very compute-intensive indexing phase and require careful parameter tuning. Therefore, exact brute force and tree-based search methods are still widely used. Here we propose a new fixed-radius near neighbor search method, called SNN, that significantly improves over brute force and tree-based methods in terms of index and query time, provably returns exact results, and requires no parameter tuning. SNN exploits a sorting of the data points by their first principal component to prune the query search space. Further speedup is gained from an efficient implementation using high-level basic linear algebra subprograms (BLAS). We provide theoretical analysis of our method and demonstrate its practical performance when used stand-alone and when applied within the DBSCAN clustering algorithm.

BLAS Kütüphanelerinin GPU Mimarilerindeki Nicel Performans Analizi

Basic Linear Algebra Subprograms (BLAS) are a set of linear algebra routines commonly used by machine learning applications and scientific computing. BLAS libraries with optimized implementations of BLAS routines offer high performance by exploiting parallel execution units in target computing systems. With massively large number of cores, graphics processing units (GPUs) exhibit high performance for computationally-heavy workloads. Recent BLAS libraries utilize parallel cores of GPU architectures efficiently by employing inherent data parallelism. In this study, we analyze GPU-targeted functions from two BLAS libraries, cuBLAS and MAGMA, and evaluate their performance on a single-GPU NVIDIA architecture by considering architectural features and limitations. We collect architectural performance metrics and explore resource utilization characteristics. Our work aims to help researchers and programmers to understand the performance behavior and GPU resource utilization of the BLAS routines implemented by the libraries.

An efficient heterogeneous parallel algorithm of the 3D MOC for multizone heterogeneous systems

Recent innovations in powerful high-performance computing (HPC) architecture have enabled high-fidelity whole-core neutron transport simulations in a reasonable amount of time. Among high-fidelity numerical methods, the one-step 3D Method of Characteristic (MOC) is becoming increasingly popular in the reactor design community due to its numerical accuracy, geometrical flexibility and high degrees of parallelism. In this work, we propose a sophisticated heterogeneous parallel algorithm of the 3D MOC based on a supercomputer prototype consisting of brand-new multizone heterogeneous processors called the MT-3000, which has a peak double precision performance of 11.6 TFLOPS when operating at 1.2 GHz. In the 3D MOC calculation procedure, the total run time is dominated by the so-called MOC transport sweep where large amounts of tracks traverse the mesh grids along different directions. We significantly improve the performance of the MOC transport sweep through a three-level parallel algorithm with multiple hiding latency techniques. We have also implemented heterogeneous level-1 basic linear algebra subprograms (BLAS) that's the second computational hot spot. The 3D C5G7 benchmark problem is employed to verify the correctness and efficiency of the heterogeneous parallel 3D MOC algorithm. Numerical results show that it achieves very high performance on the MT-3000-based supercomputer prototype. Specifically, the three-layer parallel algorithm has proven to be highly scalable for each layer, and the Quarter-MT-3000 performance of the MOC transport sweep has achieved 23.6% of the peak performance, which is close to the theoretical upper limit.

Parallel Optimization of BLAS on a New-Generation Sunway Supercomputer

The new-generation Sunway supercomputer has ultra-high computing capacity. But due to the unique heterogeneous architecture of the supercomputer, the open-source versions of basic linear algebra subprograms (BLAS) are insufficient for performance or compatibility. In addition, due to the update of the architecture, BLAS based on the previous Sunway could not fully exploit the performance of the successor. To address the challenges, we propose an optimized BLAS on the new-generation Sunway supercomputer in this paper. Specially, for achieving efficient computation, a parallel optimization method based on the new-generation Sunway for the Level-1 BLAS computing between vectors and the Level-2 BLAS computing between vectors and matrices is first proposed. Then, an adaptive scheduling algorithm for various data sizes is proposed, which is used to balance the tasks of core groups. Finally, to achieve highly efficient general matrix multiplication (GEMM) kernels, a parallel optimization method based on the new-generation Sunway for the Level-3 BLAS computing between matrices is proposed, which includes source-level optimization as well as assembly-level optimization. Experimental results show that the memory bandwidth utilization of the optimized Level-1/2 BLAS exceeds 95%, and the computational efficiency of the optimized GEMM kernel exceeds 94%.

Experiences with nested parallelism in task-parallel applications using malleable BLAS on multicore processors

Malleability is defined as the ability to vary the degree of parallelism at runtime, and is regarded as a means to improve core occupation on state-of-the-art multicore processors tshat contain tens of computational cores per socket. This property is especially interesting for applications consisting of irregular workloads and/or divergent executions paths. The integration of malleability in high-performance instances of the Basic Linear Algebra Subprograms (BLAS) is currently nonexistent, and, in consequence, applications relying on these computational kernels cannot benefit from this capability. In response to this scenario, in this paper we demonstrate that significant performance benefits can be gathered via the exploitation of malleability in a framework designed to implement portable and high-performance BLAS-like operations. For this purpose, we integrate malleability within the BLIS library, and provide an experimental evaluation of the result on three different practical use cases.

YaConv: Convolution with Low Cache Footprint

This article introduces YaConv , a new algorithm to compute convolution using GEMM microkernels from a Basic Linear Algebra Subprograms library that is efficient for multiple CPU architectures. Previous approaches either create a copy of each image element for each filter element or reload these elements into cache for each GEMM call, leading to redundant instances of the image elements in cache. Instead, YaConv loads each image element once into the cache and maximizes the reuse of these elements. The output image is computed by scattering results of the GEMM microkernel calls to the correct locations in the output image. The main advantage of this new algorithm—which leads to better performance in comparison to the existing im2col approach on several architectures—is a more efficient use of the memory hierarchy. The experimental evaluation on convolutional layers from PyTorch, along with a parameterized study, indicates an average 24% speedup over im2col convolution. Increased performance comes as a result of 3× reduction in L3 cache accesses and 2× fewer branch instructions.

An Energy-Efficient Domain-Specific Reconfigurable Array Processor With Heterogeneous PEs for Wearable Brain–Computer Interface SoCs

Recently, there is increasing demand for energy-efficient signal processing in wearable visual-stimuli-based brain-computer interface (V-BCI) devices. For the better accuracy and the reduced latency of the V-BCI system, the target identification (TI) algorithm that analyzes brain signals is being advanced, and the importance of an energy-efficient accelerating chip that processes various linear algebra operations constituting the TI algorithms is growing. In this paper, we propose a domain-specific reconfigurable array processor (RAP) with a dynamically reconfigurable and scalable array including 5-heterogeneous processing elements (PEs) for the energy-efficient acceleration of basic linear algebra subprograms (BLAS) and matrix decompositions. The system-on-chip (SoC), including the proposed RAP, was fabricated in 130-nm CMOS technology with an area of 16.87-mm2 and measured at 1.0 V 90 MHz. The RAP achieved an information transfer rate (ITR) of 139.9-bits/min and a TI accuracy of 95.4% on a fabricated chip through an optimized TI algorithm and scalable array processing. In addition, the RAP has <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$16.8\times $ </tex-math></inline-formula> higher TI energy efficiency than prior work and achieved an energy efficiency of 2144.2-bits/min/mW for information transfer processing rate with the proposed TI algorithm. The RAP supports a greater variety of linear algebra operations and data sizes with hardware reconfiguration than the prior accelerators.

An Orthogonalization-Free Parallelizable Framework for All-Electron Calculations in Density Functional Theory

All-electron calculations play an important role in density functional theory, in which improving computational efficiency is one of the most needed and challenging tasks. In the model formulations, both nonlinear eigenvalue problem and total energy minimization problem pursue orthogonal solutions. Most existing algorithms for solving these two models invoke orthogonalization process either explicitly or implicitly in each iteration. Their efficiency suffers from this process in view of its cubic complexity and low parallel scalability in terms of the number of electrons for large scale systems. To break through this bottleneck, we propose an orthogonalization-free algorithm framework based on the total energy minimization problem. It is shown that the desired orthogonality can be gradually achieved without invoking orthogonalization in each iteration. Moreover, this framework fully consists of Basic Linear Algebra Subprograms (BLAS) operations and thus can be naturally parallelized. The global convergence of the proposed algorithm is established. We also present a precondition technique which can dramatically accelerate the convergence of the algorithm. The numerical experiments on all-electron calculations show the efficiency and high scalability of the proposed algorithm.

Fast multiplicative algorithms for symmetric nonnegative tensor factorization

Symmetric nonnegative tensor factorization (SNTF) is an important tool for clustering analysis. To date, most of algorithms for SNTF are based on multiplicative update rules, which have many attractive properties, e.g., they are often simple to implement and can enforce nonnegativity without extra projection steps. However, the existing multiplicative algorithms often converge slowly due to the conservative multiplicative learning steps or the use of low-level BLAS (basic linear algebra subprograms) in their implementation. In this paper, three new multiplicative algorithms are proposed for SNTF to overcome the drawback of slow convergence. First, a parallel multiplicative algorithm, which can be implemented with high-level BLAS, is derived by auxiliary optimization. To further accelerate the convergence, two new parallel multiplicative algorithms, which enable larger learning steps for improving efficiency, are developed based on weighted geometric mean and weighted arithmetic mean, respectively. Finally, we apply the proposed algorithms to multiway probabilistic clustering, where a new hyper-stochastic normalization scheme based on Euclidean distance is developed for better data preprocessing. The experiment results on both synthetic and real-world data show that the proposed SNTF algorithms converge faster than the state-of-the-art algorithms.

Parallel time integration using Batched BLAS (Basic Linear Algebra Subprograms) routines

We present an approach for integrating the time evolution of quantum systems. We leverage the computation power of graphics processing units (GPUs) to perform the integration of all time steps in parallel. The performance boost is especially prominent for small to medium-sized quantum systems. The devised algorithm can largely be implemented using the recently-specified batched versions of the BLAS routines, and can therefore be easily ported to a variety of platforms. Our PARAllelized Matrix Exponentiation for Numerical Time evolution (PARAMENT) implementation runs on CUDA-enabled graphics processing units. Program summaryProgram Title: PARAMENTCPC Library link to program files:https://doi.org/10.17632/zy5v4xs89d.1Developer's repository link:https://github.com/parament-integrator/paramentLicensing provisions: Apache 2.0Programming language: C / CUDA / PythonNature of problem: Time-integration of the Schrödinger equation with a time-dependent Hamiltonian for quantum systems with a small Hilbert space but many time-steps.Solution method: A 4th order Magnus integrator, highly parallelized on a GPU, implemented using a small subset of BLAS functions for improved portability.

A Set of Batched Basic Linear Algebra Subprograms and LAPACK Routines

This article describes a standard API for a set of Batched Basic Linear Algebra Subprograms (Batched BLAS or BBLAS). The focus is on many independent BLAS operations on small matrices that are grouped together and processed by a single routine, called a Batched BLAS routine. The matrices are grouped together in uniformly sized groups, with just one group if all the matrices are of equal size. The aim is to provide more efficient, but portable, implementations of algorithms on high-performance many-core platforms. These include multicore and many-core CPU processors, GPUs and coprocessors, and other hardware accelerators with floating-point compute facility. As well as the standard types of single and double precision, we also include half and quadruple precision in the standard. In particular, half precision is used in many very large scale applications, such as those associated with machine learning.

TxSim: Modeling Training of Deep Neural Networks on Resistive Crossbar Systems

Deep neural networks (DNNs) have gained tremendous popularity in recent years due to their ability to achieve superhuman accuracy in a wide variety of machine learning tasks. However, the compute and memory requirements of DNNs have grown rapidly, creating a need for energy-efficient hardware. Resistive crossbars have attracted significant interest in the design of the next generation of DNN accelerators due to their ability to natively execute massively parallel vector-matrix multiplications within dense memory arrays. However, crossbar-based computations face a major challenge due to device and circuit-level nonidealities, which manifest as errors in the vector-matrix multiplications and eventually degrade DNN accuracy. To address this challenge, there is a need for tools that can model the functional impact of nonidealities on DNN training and inference. Existing efforts toward this goal are either limited to inference or are too slow to be used for large-scale DNN training. We propose TxSim, a fast and customizable modeling framework to functionally evaluate DNN training on crossbar-based hardware considering the impact of nonidealities. The key features of TxSim that differentiate it from prior efforts are: 1) it comprehensively models nonidealities during all training operations (forward propagation, backward propagation, and weight update) and 2) it achieves computational efficiency by mapping crossbar evaluations to well-optimized Basic Linear Algebra Subprograms (BLAS) routines and incorporates speedup techniques to further reduce simulation time with minimal impact on accuracy. TxSim achieves 6×- 108× improvement in simulation speed over prior works, and thereby makes it feasible to evaluate the training of large-scale DNNs on crossbars. Our experiments using TxSim reveal that the accuracy degradation in DNN training due to nonidealities can be substantial (3%-36.4%) for large-scale DNNs and data sets, underscoring the need for further research in mitigation techniques. We also analyze the impact of various device and circuit-level parameters and the associated nonidealities to provide key insights that can guide the design of crossbar-based DNN training accelerators.

Analytical modeling of matrix–vector multiplication on multicore processors

The efficiency of matrix–vector multiplication is of considerable importance. No current approaches can optimize this sufficiently well under severe time constraints. All major existing methods are based on either manual‐tuning or auto‐tuning and can therefore be time‐consuming. We introduce an alternative model‐driven approach, which is used to map the implementation of matrix–vector multiplication to a target architecture and analytically obtain its parameters. The approach yields the performance that is competitive with optimized Basic Linear Algebra Subprograms (BLAS)‐like dense linear algebra libraries without the need for manual‐tuning or auto‐tuning. Our method provides competitive performance across hardware architectures and can be utilized to obtain single‐threaded and multi‐threaded implementations on multicore processors. We expect that this approach allows the community to progress from valuable engineering solutions to techniques with a broader application.

Algorithm 1005

A set of Fortran subroutines for reverse mode algorithmic (or automatic) differentiation of the basic linear algebra subprograms (BLAS) is presented. This is preceded by a description of the mathematical tools used to obtain the formulae of these derivatives, with emphasis on special matrices supported by the BLAS: triangular, symmetric, and band. All single and double precision BLAS derivatives have been implemented, together with the Cholesky factorization from Linear Algebra Package (LAPACK). The subroutines are written in Fortran 2003 with a Fortran 77 interface to allow use from C and C++, as well as dynamic languages such as R, Python, Matlab, and Octave. The subroutines are all implemented by calling BLAS, thereby attaining fast runtime. Timing results show derivative runtimes that are about twice those of the corresponding BLAS, in line with theory. The emphasis is on reverse mode because it is more important for the main application that we have in mind, numerical optimization. Two examples are presented, one dealing with the least squares modeling of groundwater, and the other dealing with the maximum likelihood estimation of the parameters of a vector autoregressive time series. The article contains comprehensive tables of formulae for the BLAS derivatives as well as for several non-BLAS matrix operations commonly used in optimization.

An Error-Free Transformation for Matrix Multiplication with Reproducible Algorithms and Divide and Conquer Methods

This paper discusses accurate numerical algorithms for matrix multiplication. Matrix multiplication is a basic and important problem in numerical linear algebra. Numerical computations using floating-point arithmetic can be quickly performed on existing computers. However, the accumulation of rounding errors due to finite precision arithmetic is a critical problem. An error-free transformation for matrix multiplication is reviewed in this paper. Such a transformation is extremely useful for developing accurate numerical algorithms for matrix multiplication. One advantage of the transformation is that it exploits Basic Linear Algebra Subprograms (BLAS). We provide a rounding error analysis of reproducible algorithms for matrix multiplication based on the error-free transformation. In addition, we propose an error-free transformation for matrix multiplication that can be utilized with the divide and conquer methods.

Design and implementation of multiple-precision BLAS Level 1 functions for graphics processing units

Basic Linear Algebra Subprograms (BLAS) are the building blocks for various numerical algorithms and are widely used in scientific computations. However, some linear algebra applications need more precision than the standard double precision available in most existing BLAS libraries. In this paper, we implement and evaluate multiple-precision scalar and vector BLAS functions on graphics processing units (GPUs). We use the residue number system (RNS) to represent arbitrary length floating-point numbers. The non-positional nature of RNS enables parallelism in multiple-precision arithmetic and makes RNS a good tool for high-performance computing applications. We first present new data-parallel algorithms for multiplying and adding RNS-based floating-point representations. Next, we suggest algorithms for multiple-precision vectors specially designed for parallel computations on GPUs. Using these algorithms, we develop and evaluate four GPU-accelerated multiple-precision BLAS functions, ASUM, DOT, SCAL, and AXPY. It is shown through experiments that in many cases, the implemented functions achieve significantly better performance compared to existing multiple-precision software for CPU and GPU.

FT-PBLAS: PBLAS-Based Fault-Tolerant Linear Algebra Computation on High-performance Computing Systems

As high-performance computing (HPC) systems have scaled up, resilience has become a great challenge. To guarantee resilience, various kinds of hardware and software techniques have been proposed. However, among popular software fault-tolerant techniques, both the checkpoint-restart approach and the replication technique face challenges of scalability in the era of peta- and exa-scale systems due to their numerous processes. In this situation, algorithm-based approaches, or algorithm-based fault tolerance (ABFT) mechanisms, have become attractive because they are efficient and lightweight. Although the ABFT technique is algorithm-dependent, it is possible to implement it at a low level (e.g., in libraries for basic numerical algorithms) and make it application-independent. However, previous ABFT approaches have mainly aimed at achieving fault tolerance in integrated circuits (ICs) or at the architecture level and are therefore not suitable for HPC systems; e.g., they use checksums of rows and columns of matrices rather than checksums of blocks to detect errors. Furthermore, they cannot deal with errors caused by node failure, which are common in current HPC systems. To solve these problems, this paper proposes FT-PBLAS, a PBLAS-based library for fault-tolerant parallel linear algebra computations that can be regarded as a fault-tolerant version of the parallel basic linear algebra subprograms (PBLAS), because it provides a series of fault-tolerant versions of interfaces in PBLAS. To support the underlying error detection and recovery mechanisms in the library, we propose a block-checksum approach for non-fatal errors and a scheme for addressing node failure, respectively. We evaluate two fault-tolerant mechanisms and FT-PBLAS on HPC systems, and the experimental results demonstrate the performance of our library.

Integration and exploitation of intra-routine malleability in BLIS

Malleability is a property of certain applications (or tasks) that, given an external request or autonomously, can accommodate a dynamic modification of the degree of parallelism being exploited at runtime. Malleability improves resource usage (core occupation) on modern multicore architectures for applications that exhibit irregular and divergent execution paths and heavily depend on the underlying library performance to attain high performance. The integration of malleability within high-performance instances of the Basic Linear Algebra Subprograms (BLAS) is nonexistent, and, in addition, it is difficult to attain given the rigidity of current application programming interfaces (APIs). In this paper, we overcome these issues presenting the integration of a malleability mechanism within BLIS, a high-performance and portable framework to implement BLAS-like operations. For this purpose, we leverage low-level (yet simple) APIs to integrate on-demand malleability across all Level-3 BLAS routines, and we demonstrate the performance benefits of this approach by means of a higher-level dense matrix operation: the LU factorization with partial pivoting and look-ahead.

2.1 ANALYSES OF THE BLOC AND BLAS RATING SCALES AND THEIR IMPACT ON ADOLESCENTS AGE 11–17 DIAGNOSED WITH OCD AND/OR ANXIETY DISORDER

Standardized rating scales are commonly used in mental health care. Several scales used for adolescents diagnosed with anxiety and/or OCD include Patient-Reported Outcomes Measurement Information System (PROMIS), Screen for Child Anxiety Related Disorders (SCARED), and Yale-Brown Obsessive-Compulsive Scale (Y-BOCS). Lin Lu, MD, at Rosalind Franklin University developed “blink light rating scales” (Basic Linear Algebra Subprograms [BLAS] and blink light obsessive-compulsive [BLOC]), which have patients consider their anxious thoughts as a blinking light. Patients rate on a scale (0–4) of how much light interferes with activities. This project’s purpose is to examine whether BLAS (blink light anxiety) and BLOC scales are useful to patients compared with standardized rating scales.

SCELib4.0: The new program version for computing molecular properties in the Single Center Approach

SCELib4.0: The new program version for computing molecular properties in the Single Center Approach