Dot-product Computation Research Articles

Multiple-base Logarithmic Quantization and Application in Reduced Precision AI Computations

The power of logarithmic quantizations and computations has been recognized as a useful tool in optimizing the performance of large ML models. There are plenty of applications of ML techniques in digital preservation. The accuracy of computations may play a crucial role in the corresponding algorithms. In this article, we provide results that demonstrate significantly better quantization signal-to-noise ratio performance thanks to multiple-base logarithmic number systems (MDLNS) in comparison with the floating point quantizations that use the same number of bits. On a hardware level, we present details about our Xilinx VCU-128 FPGA design for dot product and matrix vector computations. The MDLNS matrix-vector design significantly outperforms equivalent fixed-point binary designs in terms of area (A) and time (T) complexity and power consumption as evidenced by a 4 × scaling of AT 2 metric for VLSI performance, and 57% increase in computational throughput per watt compared to fixed-point arithmetic.

High-efficiency Compressor Trees for Latest AMD FPGAs

High-fan-in dot product computations are ubiquitous in highly relevant application domains, such as signal processing and machine learning. Particularly, the diverse set of data formats used in machine learning poses a challenge for flexible efficient design solutions. Ideally, a dot product summation is composed from a carry-free compressor tree followed by a terminal carry-propagate addition. On FPGA, these compressor trees are constructed from generalized parallel counters whose architecture is closely tied to the underlying reconfigurable fabric. This work reviews known counter designs and proposes new ones in the context of the new AMD Versal™ fabric. On this basis, we develop a compressor generator featuring variable-sized counters, novel counter composition heuristics, explicit clustering strategies, and case-specific optimizations like logic gate absorption. In comparison to the Vivado™ default implementation, the combination of such a compressor with a novel, highly efficient quaternary adder reduces the LUT footprint across different bit matrix input shapes by 45% for a plain summation and by 46% for a terminal accumulation at a slight cost in critical path delay still allowing an operation well above 500 MHz. We demonstrate the aptness of our solution at examples of low-precision integer dot product accumulation units.

An advanced airport terminal cooling load forecasting model integrating SSA and CNN-Transformer

The cooling system in airport terminals experiences dynamic fluctuations of cooling load, influenced by factors like outdoor environmental climate and indoor passenger flow. Therefore, establishing a precise and reliable cooling load forecast is crucial for the energy-efficient operation of the cooling system. This paper aims to develop a model for forecasting cooling loads in airport terminals, providing accurate guidelines for energy-saving and optimal operations. Firstly, Singular Spectrum Analysis is applied for data reconstruction. Secondly, a one-dimensional model with a Convolutional Neural Network Algorithm is integrated, replacing the conventional sine and cosine positional encoding. Finally, recognizing the traditional Transformer's limited responsiveness to local features, this study adopts a causal Convolutional approach as an alternative to dot product computation, followed by empirical validation using air conditioning load data from airport terminals. The experimental results demonstrate mean absolute percentage error of 2.16%, 6.39% and 10.71% for single-step, two-step and three-step predictions, respectively, confirming the reliability of the model. It demonstrates superior accuracy compared to various models and provides crucial insights for predicting cooling loads in large public structures with similar data complexity. This model can serve as a solied foundation for guiding the cooling system of airport terminals toward the objectives of energy conservation and emissions reduction.

TransInver: 3D data-driven seismic inversion based on self-attention

Recently, convolutional neural network (CNN)-based deep learning (DL) for impedance inversion has been extended to multiple dimensions. Training multidimensional DL inversion requires extracting supervised information from sparse 1D well-log labels. Fully convolutional networks rely on their parameter sharing mechanism and receptive fields to achieve this, but their perceptual range is limited, making it difficult to capture long-term correlations in seismic data. The transformer is a type of network that is entirely based on self-attention, and it has demonstrated remarkable performance across various tasks and domains. However, its suitability for 3D seismic inversion is restricted by its high computational workload, fixed input size requirement, and inadequate handling of low-level details. The primary goal was to reengineer the self-attention mechanism to optimize its applicability for seismic impedance inversion tasks. The high-dimensional self-attention is decoupled into dual low-dimensional attention paths to reduce the computation of dense connections and matrix dot products. Shared parameters are used instead of full connection, allowing for flexible changes in input sizes by the network. In addition, its local modeling capabilities are enhanced by integrating it with the residual structure of the CNN. We name the resulting structure Self-Attention ResBlock, which is used as the basic unit for constructing TransInver. Comparative experiments indicate that TransInver performs significantly better than 3D methods such as UNet, TransUNet, HRNet, and 1D inversion methods. TransInver produces reliable inversion results using only nine well logs for SEAM Phase I and three well logs for the field data set of the Netherlands F3. This backbone network can deliver excellent inversion performance without depending on any auxiliary means such as low-frequency constraints or semisupervised frameworks.

Evidence of Transport Degradation in 22 nm FD-SOI Charge Trapping Transistors for Neural Network Applications

This article reports on the characterization and analysis of 22 nm FD-SOI CMOS technology-based charge trap transistors (CTT) and their application in neural networks. The working principle of CTT as non-volatile memory depends on trapping and de-trapping of charge in the high-k gate dielectric with the application of voltage pulses on the drain and gate terminals. This programming condition leads to the generation of interfacial traps that can drastically impact device performance. We report degradation in CTT through extractions of subthreshold swing, ON-state current, and mobility. We elucidate transport degradation by correlating effective mobility to interface trap density using a quasi-ballistic transport theory essential for the nanoscale devices under test in this work. Importantly, the trap-induced degradation in CTT transport is not recovered with voltage pulsing at room temperature as our experiments show. We demonstrate the implications of these findings based on analog computation of dot products, an operation of utmost importance to the implementation of artificial neural networks.

Memristive-based Mixed-signal CGRA for Accelerating Deep Neural Network Inference

In this paper, a mixed-signal coarse-grained reconfigurable architecture (CGRA) for accelerating inference in deep neural networks (DNNs) is presented. It is based on performing dot-product computations using analog computing to achieve a considerable speed improvement. Other computations are performed digitally. In the proposed structure (called MX-CGRA), analog tiles consisting of memristor crossbars are employed. To reduce the overhead of converting the data between analog and digital domains, we utilize a proper interface between the analog and digital tiles. In addition, the structure benefits from an efficient memory hierarchy where the data is moved as close as possible to the computing fabric. Moreover, to fully utilize the tiles, we define a set of micro instructions to configure the analog and digital domains. Corresponding context words used in the CGRA are determined by these instructions (generated by a companion compiler tool). The efficacy of the MX-CGRA is assessed by modeling the execution of state-of-the-art DNN architectures on this structure. The architectures are used to classify images of the ImageNet dataset. Simulation results show that, compared to the previous mixed-signal DNN accelerators, on average, a higher throughput of 2.35 × is achieved.

Dot-product computation and logistic regression with 2D hexagonal-boron nitride (h-BN) memristor arrays

This work reports on the hardware implementation of analog dot-product operation on arrays of two-dimensional (2D) hexagonal boron nitride (h-BN) memristors. This extends beyond previous work that studied isolated device characteristics towards the application of analog neural network accelerators based on 2D memristor arrays. The wafer-level fabrication of the memristor arrays is enabled by large-area transfer of CVD-grown few-layer (8 layers) h-BN films. Individual devices achieve an on/off ratio of >10, low voltage operation (∼0.5 V set/V reset), good endurance (>6000 programming steps), and good retention (>104 s). The dot-product operation shows excellent linearity and repeatability, with low read energy consumption (∼200 aJ to 20 fJ per operation), with minimal error and deviation over various measurement cycles. Moreover, we present the implementation of a stochastic logistic regression algorithm in 2D h-BN memristor hardware for the classification of noisy images. The promising resistive switching characteristics, performance of dot-product computation, and successful demonstration of logistic regression in h-BN memristors signify an important step towards the integration of 2D materials for next-generation neuromorphic computing systems.

Bridge Memristor Super-Resolution Crossbars

A higher number of unique conductance levels in the memristor crossbar node ensures higher accuracy for analogue dot-product computations. Series and parallel combinations of memristors are used as a common approach to creating multilevel conductance superresolution nodes. The super-resolution nodes are tolerant to a wide range of memristor non-idealities. In this paper, we present a bridge-type design of a super-resolution node. Compared to previous work, the proposed design could achieve a resolution improvement of 5 to 9 times higher when individual memristor conductance levels change from 5 to 12 states. Although the bridge nodes preserve tolerance to memristor variability similar to that of a parallel superresolution node, they do so at lower area and power requirements. Bridge-based crossbar nodes were successfully tested in artificial and convolutional neural networks, showing near-idealistic inference accuracy.

STeP-CiM: Strain-Enabled Ternary Precision Computation-In-Memory Based on Non-Volatile 2D Piezoelectric Transistors

We proposed 2D piezoelectric FET (PeFET)–based compute-enabled non-volatile memory for ternary deep neural networks (DNNs). PeFETs hinge on ferroelectricity for bit storage and piezoelectricity for bit sensing, exhibiting inherently amenable features for computation-in-memory of dot products of weights and inputs in the signed ternary regime. PeFETs consist of a material with ferroelectric and piezoelectric properties coupled with a transition metal dichalcogenide channel. We utilized (a) ferroelectricity to store binary bits (0/1) in the form of polarization (−P/+P) and (b) polarization-dependent piezoelectricity to read the stored state by means of strain-induced bandgap change in the transition metal dichalcogenide channel. The unique read mechanism of PeFETs enables us to expand the traditional association of +P (−P) with low (high) resistance states to their dual high (low) resistance depending on read voltage. Specifically, we demonstrated that +P (−P) stored in PeFETs can be dynamically configured in (a) a low (high) resistance state for positive read voltages and (b) their dual high (low) resistance states for negative read voltages, without afflicting a read disturb. Such a feature, which we named as polarization-preserved piezoelectric effect reversal with dual voltage polarity (PiER), is unique to PeFETs and has not been shown in hitherto explored memories. We leveraged PiER to propose a Strain-enabled Ternary Precision Computation-in-Memory (STeP-CiM) cell with capabilities of computing the scalar product of the stored weight and input, both of which are represented with signed ternary precision. Furthermore, using multi-word line assertion of STeP-CiM cells, we achieved massively parallel computation of dot products of signed ternary inputs and weights. Our array-level analysis showed 91% lower delay and improvements of 15% and 91% in energy for in-memory multiply-and-accumulate operations compared to near-memory design approaches based on 2D FET–based SRAM and PeFET, respectively. We also analyzed the system-level implications of STeP-CiM by deploying it in a ternary DNN accelerator. STeP-CiM exhibits 6.11× to 8.91× average improvement in performance and 3.2× average improvement in energy over SRAM-based near-memory design. We also compared STeP-CiM to near-memory design based on PeFETs showing 5.67× to 6.13× average performance improvement and 6.07× average energy savings.

Enabling In-Memory Computations in Non-Volatile SRAM Designs

The rapid growth in development of neural networks has necessitated the requirement of large capacity on-chip SRAM’s for Machine Learning accelerators. This has resulted in SRAM’s occupying significant portion of the die area. Furthermore, due to increased short channel effects in advanced CMOS technology nodes, the Vt of the transistors are increased to reduce the leakage power effectively. Vt increase results in direct increase in V <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MIN</sub> (minimum operating voltage) of the device. The conventional 6T SRAM with the use of RRAM(R) to store the bitcell storage node values and PTM(S) as a selector device (6T-2R-2S) can help in decoupling V <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MIN</sub> and Vt requirement with minimum area overhead. Functionalities of 6T-2R-2S bitcell are investigated to present a 2T-2R-2S mode and SRAM-RRAM hybrid mode of operation, further utilized in performing Compute in Memory(CIM). The above functionalities can be presented in a 8T2R bitcell, that makes use of transistor in place of PTM. 2T-2R-2S/2T-2R mode is a fully non-differential mode of operation, leveraging only the NVM portion of the bitcell. Furthermore, SRAM-RRAM hybrid mode is proposed making use of the SRAM read port transistors to perform read operation of the data stored on the RRAM, during standby mode. The architecture study of set-associative cache made of 6T-2R-2S array is also presented. The 2T-2R-2S/2T-2R mode coupled with SRAM-only mode can be efficiently used to perform CIM for dot product and XNOR computation with co-locating the weights and activations stored onto the same bitcell. System level study highlighting energy efficiency and speedup along with the proposed CIM architecture’s analysis on CIFAR-10 dataset is presented. Design sensitivity analysis with respect to PTM and RRAM parameters is discussed for both the bitcells.

Non-volatile memory devices offer alternative computer architecture for neural networks

Improvements can lead to spiking neural networks with significantly higher storage density and in-memory, highly parallel dot product computation.

ERA-LSTM: An Efficient ReRAM-Based Architecture for Long Short-Term Memory

Processing-in-memory (PIM) architecture based on resistive random access memory (ReRAM) crossbars is a promising solution to the memory bottleneck that long short-term memory (LSTM) faces. Based on the dataflow analysis of the LSTM computing paradigm, this article proposes to adopt the ReRAM-based analog approximate computing to conduct the LSTM-specific element-wise computation. Combined with the dot-product computation implemented with ReRAM crossbars, a new LSTM processing tile is designed to significantly reduce the demand for analog-to-digital converters (ADCs), which is the major part of power consumption of existing designs. Next, we elaborate on a mapping scheme to efficiently deploy large-scale LSTM onto multiple processing tiles. Finally, an architecture enhancement is proposed to support crossbar-friendly LSTM pruning to further improve efficiency. This overall design, named ERA-LSTM, is presented. Our evaluation shows that it can outperform two state-of-the-art FPGA-based LSTM accelerators by 103.6 and 35.9 times, respectively; compared with a state-of-the-art ReRAM-based LSTM accelerator with digital element-wise computation, it is 6.1 times more efficient. Moreover, our experiments demonstrate that the impact of hardware constraints and approximation errors on the inference accuracy can be effectively reduced by the proposed fine-tuning scheme and by optimizing the design of the approximator.

Pathways to efficient neuromorphic computing with non-volatile memory technologies

Historically, memory technologies have been evaluated based on their storage density, cost, and latencies. Beyond these metrics, the need to enable smarter and intelligent computing platforms at a low area and energy cost has brought forth interesting avenues for exploiting non-volatile memory (NVM) technologies. In this paper, we focus on non-volatile memory technologies and their applications to bio-inspired neuromorphic computing, enabling spike-based machine intelligence. Spiking neural networks (SNNs) based on discrete neuronal “action potentials” are not only bio-fidel but also an attractive candidate to achieve energy-efficiency, as compared to state-of-the-art continuous-valued neural networks. NVMs offer promise for implementing both area- and energy-efficient SNN compute fabrics at almost all levels of hierarchy including devices, circuits, architecture, and algorithms. The intrinsic device physics of NVMs can be leveraged to emulate dynamics of individual neurons and synapses. These devices can be connected in a dense crossbar-like circuit, enabling in-memory, highly parallel dot-product computations required for neural networks. Architecturally, such crossbars can be connected in a distributed manner, bringing in additional system-level parallelism, a radical departure from the conventional von-Neumann architecture. Finally, cross-layer optimization across underlying NVM based hardware and learning algorithms can be exploited for resilience in learning and mitigating hardware inaccuracies. The manuscript starts by introducing both neuromorphic computing requirements and non-volatile memory technologies. Subsequently, we not only provide a review of key works but also carefully scrutinize the challenges and opportunities with respect to various NVM technologies at different levels of abstraction from devices-to-circuit-to-architecture and co-design of hardware and algorithm.

8T SRAM Cell as a Multibit Dot-Product Engine for Beyond Von Neumann Computing

Large-scale digital computing almost exclusively relies on the von Neumann architecture, which comprises separate units for storage and computations. The energy-expensive transfer of data from the memory units to the computing cores results in the well-known von Neumann bottleneck. Various approaches aimed toward bypassing the von Neumann bottleneck are being extensively explored in the literature. These include in-memory computing based on CMOS and beyond CMOS technologies, wherein by making modifications to the memory array, vector computations can be carried out as close to the memory units as possible. Interestingly, in-memory techniques based on CMOS technology are of special importance due to the ubiquitous presence of field-effect transistors and the resultant ease of large-scale manufacturing and commercialization. On the other hand, perhaps the most important computation required for applications such as machine learning, etc., comprises the dot-product operation. Emerging nonvolatile memristive technologies have been shown to be very efficient in computing analog dot products in an in situ fashion. The memristive analog computation of the dot product results in much faster operation as opposed to digital vector in-memory bitwise Boolean computations. However, challenges with respect to large-scale manufacturing coupled with the limited endurance of memristors have hindered rapid commercialization of memristive-based computing solutions. In this paper, we show that the standard 8 transistor (8T) digital SRAM array can be configured as an analoglike in-memory multibit dot-product engine (DPE). By applying appropriate analog voltages to the read ports of the 8T SRAM array and sensing the output current, an approximate analog–digital DPE can be implemented. We present two different configurations for enabling multibit dot-product computations in the 8T SRAM cell array, without modifying the standard bit-cell structure. We also demonstrate the robustness of the present proposal in presence of nonidealities such as the effect of line resistances and transistor threshold voltage variations. Since our proposal preserves the standard 8T-SRAM array structure, it can be used as a storage element with standard read–write instructions and also as an on-demand analoglike dot-product accelerator.

CONV-SRAM: An Energy-Efficient SRAM With In-Memory Dot-Product Computation for Low-Power Convolutional Neural Networks

This paper presents an energy-efficient static random access memory (SRAM) with embedded dot-product computation capability, for binary-weight convolutional neural networks. A 10T bit-cell-based SRAM array is used to store the 1-b filter weights. The array implements dot-product as a weighted average of the bitline voltages, which are proportional to the digital input values. Local integrating analog-to-digital converters compute the digital convolution outputs, corresponding to each filter. We have successfully demonstrated functionality (>98% accuracy) with the 10 000 test images in the MNIST hand-written digit recognition data set, using 6-b inputs/outputs. Compared to conventional full-digital implementations using small bitwidths, we achieve similar or better energy efficiency, by reducing data transfer, due to the highly parallel in-memory analog computations.

Memristor Crossbar Tiles in a Flexible, General Purpose Neural Processor

Memristor crossbar arrays have been proposed for use in synthetic neuron hardware due to their high density, non-volatile programmable conductances, and ability to perform dot-product computation efficiently. However, fitting complete neural networks into multiple, uniformly sized crossbars often results in poor memristor and neuron utilization. In this paper, we propose the use of smaller crossbar tiles that can be flexibly combined to create a variety of crossbar sizes and aspect ratios to more closely fit the ideal sizes required. We examine the throughput/power and throughput/area metrics of custom versus tiled designs for three different neural network applications, representing widely differing design points. We demonstrate that a general purpose neural processor, using a single tile size, compares reasonably with highly specialized neural processor designs using multiple, optimally sized custom crossbars. These results indicate that general purpose memristive crossbar-based neural processors are practical.

Efficient privacy‐preserving dot‐product computation for mobile big data

Many mobile big data applications require the computation of dot-product of two vectors. For examples, the dot-product of an individual's genome data collected by a body area network and the gene biomarkers of a health centre can help detect diseases in m-Health, and that of the interests of two persons can facilitate profile matching in mobile social networks. Nevertheless, mobile big data typically contain sensitive personal information and are more accessible to the general public as they are collected by mobile devices carried by human beings. Therefore exposing the inputs of dot-product computation discloses sensitive information about the two participants, leading to severe privacy violations. The authors tackle the problem of private dot-product computation targeting mobile big data applications in which secure channels are hardly established, and the computational efficiency is highly desirable. We first propose two basic schemes and then present the corresponding advanced versions to improve computational efficiency and enhance the privacy-protection strength. Furthermore, we theoretically prove that our proposed schemes can simultaneously achieve privacy-preservation, non-repudiation, and accountability. Our numerical results verify the performance of the proposed schemes in terms of communication and computational overheads.

LightSpMV: Faster CUDA-Compatible Sparse Matrix-Vector Multiplication Using Compressed Sparse Rows

Compressed sparse row (CSR) is one of the most frequently used sparse matrix storage formats. However, the efficiency of existing CUDA-compatible CSR-based sparse matrix vector multiplication (SpMV) implementations is relatively low. We address this issue by presenting LightSpMV, a parallelized CSR-based SpMV implementation programmed in CUDA C++. This algorithm achieves high speed by employing atomic and warp shuffle instructions to implement fine-grained dynamic distribution of matrix rows over vectors/warps as well as efficient vector dot product computation. Moreover, we propose a unified cache hit rate computation approach to consistently investigate the caching behavior for different SpMV kernels, which may have different data deployment in the hierarchical memory space of CUDA-enabled GPUs. We have assessed LightSpMV using a set of sparse matrices and further compared it to the CSR-based SpMV kernels in the top-performing CUSP, ViennaCL and cuSPARSE libraries. Our experimental results demonstrate that LightSpMV is superior to CUSP, ViennaCL and cuSPARSE on the same Kepler-based Tesla K40c GPU, running up to 2.63× and 2.65× faster than CUSP, up to 2.52× and 1.96× faster than ViennaCL, and up to 1.94× and 1.79× faster than cuSPARSE with respect to single and double precision, respectively. In addition, for the acceleration of the PageRank graph application, LightSpMV still keeps consistent superiority to the aforementioned three counterparts. LightSpMV is open-source and publicly available at http://lightspmv.sourceforge.net.

Explicit Recursive and Adaptive Filtering in Reproducing Kernel Hilbert Spaces

This brief presents a methodology to develop recursive filters in reproducing kernel Hilbert spaces. Unlike previous approaches that exploit the kernel trick on filtered and then mapped samples, we explicitly define the model recursivity in the Hilbert space. For that, we exploit some properties of functional analysis and recursive computation of dot products without the need of preimaging or a training dataset. We illustrate the feasibility of the methodology in the particular case of the γ-filter, which is an infinite impulse response filter with controlled stability and memory depth. Different algorithmic formulations emerge from the signal model. Experiments in chaotic and electroencephalographic time series prediction, complex nonlinear system identification, and adaptive antenna array processing demonstrate the potential of the approach for scenarios where recursivity and nonlinearity have to be readily combined.

COMPUTING DOT -- PRODUCT ON HETEROGENEOUS MASTER -- WORKER PLATFORMS

This paper is focused on designing two parallel dot product imple- mentations for heterogeneous master-worker platforms. These implementations are based on the data allocation and dynamic load balancing strategies. The first implementation is the dynamic master - worker with allocation of vectors where the master distributes vectors (data) and computations to the workers whereas the second one is the dynamic master - worker with allocation of vector pointers where the vectors are supposed to be replicated among participating resources beforehand and the master distributes computations to the workers. We also report a set of numerical experiments on a heterogeneous platform where computational resources have different computing powers. Also, the workers are connected to the master by links of same capacities. The obtained experimental results demonstrate that the dynamic allocation of vector point- ers achieve better performance than the original one for computing dot product computation. The paper also presents and verifies an accurate timing model to predict the performance of the proposed implementations on clusters of heterogeneous workstations. Through this model the viability of the proposed implementa-