Machine Learning Accelerators Research Articles

DPCrypto: Acceleration of Post-Quantum Cryptography Using Dot-Product Instructions on GPUs

Modern NVIDIA GPU architectures offer dot-product instructions (DP2A and DP4A), with the aim of accelerating machine learning and scientific computing applications. These dot-product instructions allow the computation of multiply-and-add instructions in a single clock cycle, effectively achieving higher throughput compared to conventional 32-bit integer units. In this paper, we show that the dot-product instruction can also be used to accelerate matrix-multiplication and polynomial convolution operations, which are widely used in post-quantum lattice-based cryptographic schemes. In particular, we propose a highly optimized implementation of FrodoKEM wherein the matrix-multiplication is accelerated by the dot-product instruction. We also present specially designed data structures that allow an efficient implementation of Saber key-encapsulation mechanism, utilizing the dot-product instruction to speed-up the polynomial convolution. The proposed FrodoKEM implementation achieves <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$4.37\times $ </tex-math></inline-formula> higher throughput than the state-of-the-art implementation on a V100 GPU. This paper also presents the first implementation of Saber on GPU platforms, achieving 124,418, 120,463, and 31,658 key exchanges per second on RTX3080, V100, and T4 GPUs, respectively. Since matrix-multiplication and polynomial convolution operations are the most time-consuming operations in lattice-based cryptographic schemes, we strongly believe that the proposed methods can be beneficial to other KEM and signatures schemes based on lattices.

Demonstration of accelerating machine learning inference queries with correlative proxy models

We will demonstrate a prototype query-processing engine, which utilizes correlations among predicates to accelerate machine learning (ML) inference queries on unstructured data. Expensive operators such as feature extractors and classifiers are deployed as user-defined functions (UDFs), which are not penetrable by classic query optimization techniques such as predicate push-down. Recent optimization schemes (e.g., Probabilistic Predicates or PP) build a cheap proxy model for each predicate offline, and inject proxy models in the front of expensive ML UDFs under the independence assumption in queries. Input records that do not satisfy query predicates are filtered early by proxy models to bypass ML UDFs. But enforcing the independence assumption may result in sub-optimal plans. We use correlative proxy models to better exploit predicate correlations and accelerate ML queries. We will demonstrate our query optimizer called CORE, which builds proxy models online, allocates parameters to each model, and reorders them. We will also show end-to-end query processing with or without proxy models.

Hybrid Accumulator Factored Systolic Array for Machine Learning Acceleration

Deep learning applications have become ubiquitous in today’s era and it has led to vast development in machine learning (ML) accelerators. Systolic arrays have been a primary part of ML accelerator architecture. To fully leverage the systolic arrays, it is required to explore the computer arithmetic data-path components and their tradeoffs in accelerators. We present a novel factored systolic array (FSA) architecture, in which the carry propagation adder (CPA) and carry-save adder (CSA) perform hybrid accumulation on least significant bit (LSB) bits and most significant bits (MSB) bits, respectively, inside each processing element. In addition, a small CPA to complete accumulation for MSB bits along with rounding logic for each column of the array is placed, which not only reduces the area, delay, and power but also balances the combinational and sequential area tradeoffs. We demonstrate the hybrid accumulator with partial CPA factoring in “Gemmini,” an open-source practical systolic array accelerator and factoring technique does not change the functionality of the base design. We implemented three baselines, original Gemmini and two variants of it, and show that the proposed approach leads to overall significant reduction in area within the range 12.8% – 50.2% and in power within the range 18.6% – 41% with improved or similar delay in comparison to the baselines.

OpenMDS: An Open-Source Shell Generation Framework for High-Performance Design on Xilinx Multi-Die FPGAs

FPGA is a promising platform in designing hardware due to its design flexibility and fast development cycle, despite the device's limited hardware resources. To address this, latest FPGAs have adopted a multi-die architecture that employs multiple dies in a single device to provide abundant hardware resources. However, the multi-die architecture causes critical timing issues when signal paths cross the die-to-die boundaries, adding another design challenge in using FPGA. We propose OpenMDS, an open-source shell generation framework for high-performance design on Xilinx multi-die FPGAs. Based on the user's design requirements, it generates an optimized shell for the target FPGA via die-level kernel encapsulation, automated bus pipelining, and customized floorplanning. To evaluate our shell generation, we compare its implementation results against Xilinx's Vitis framework. As a result, OpenMDS uses average 20% less logic resources than Vitis for the same shell functionality. To show its practicality, we use OpenMDS for the design of machine learning accelerator that contains multiple systolic-array processors. OpenMDS achieves 247MHz and 235MHz kernel frequency and 400MHz and 429MHz memory bus frequency for U50 and U280, respectively, for the accelerator design over 90% logic utilization, claiming up to 12.27% and 22.92% higher kernel and memory bus frequency over Vitis.

Microstructural Black Swans

Black swans are a metaphor for rare events with extreme consequences. In the domain of structural materials, black swans represent features in the microstructure that lead to catastrophic failure; as a result of their rarity, they are difficult to observe and often overlooked. These unusual weakest-link features are described variously as incipient, emergent, or anomalous. They give rise to localization, percolation, or avalanche events such as fracture, ductile rupture, dielectric breakdown, corrosion pit nucleation, and fatigue-crack initiation; as such, they are limiting cases in the concept of a representative volume. In this perspective, three examples are given of rare microstructural features and how they limit the mechanical reliability of structural metals. After taking stock of these examples, a future outlook considers the need for high-throughput testing and non-destructive characterization as well as detection algorithms and materials modelling strategies, including accelerated machine learning methods, that can capture anomalous events.

A Survey on Machine Learning Accelerators and Evolutionary Hardware Platforms

Advanced computing systems have long been enablers for breakthroughs in artificial intelligence (AI) and machine learning (ML) algorithms, either through sheer computational power or form-factor miniaturization. However, as AI/ML algorithms become more complex and the size of data sets increases, existing computing platforms are no longer sufficient to bridge the gap between algorithmic innovation and hardware design. This article presents a survey about various ML accelerators.

Optimizing machine learning inference queries with correlative proxy models

We consider accelerating machine learning (ML) inference queries on unstructured datasets. Expensive operators such as feature extractors and classifiers are deployed as user-defined functions (UDFs), which are not penetrable with classic query optimization techniques such as predicate push-down. Recent optimization schemes (e.g., Probabilistic Predicates or PP) assume independence among the query predicates, build a proxy model for each predicate offline, and rewrite a new query by injecting these cheap proxy models in the front of the expensive ML UDFs. In such a manner, unlikely inputs that do not satisfy query predicates are filtered early to bypass the ML UDFs. We show that enforcing the independence assumption in this context may result in sub-optimal plans. In this paper, we propose CORE, a query optimizer that better exploits the predicate correlations and accelerates ML inference queries. Our solution builds the proxy models online for a new query and leverages a branch-and-bound search process to reduce the building costs. Results on three real-world text, image and video datasets show that CORE improves the query throughput by up to 63% compared to PP and up to 80% compared to running the queries as it is.

Enabling In-Memory Computations in Non-Volatile SRAM Designs

The rapid growth in development of neural networks has necessitated the requirement of large capacity on-chip SRAM’s for Machine Learning accelerators. This has resulted in SRAM’s occupying significant portion of the die area. Furthermore, due to increased short channel effects in advanced CMOS technology nodes, the Vt of the transistors are increased to reduce the leakage power effectively. Vt increase results in direct increase in V <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MIN</sub> (minimum operating voltage) of the device. The conventional 6T SRAM with the use of RRAM(R) to store the bitcell storage node values and PTM(S) as a selector device (6T-2R-2S) can help in decoupling V <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MIN</sub> and Vt requirement with minimum area overhead. Functionalities of 6T-2R-2S bitcell are investigated to present a 2T-2R-2S mode and SRAM-RRAM hybrid mode of operation, further utilized in performing Compute in Memory(CIM). The above functionalities can be presented in a 8T2R bitcell, that makes use of transistor in place of PTM. 2T-2R-2S/2T-2R mode is a fully non-differential mode of operation, leveraging only the NVM portion of the bitcell. Furthermore, SRAM-RRAM hybrid mode is proposed making use of the SRAM read port transistors to perform read operation of the data stored on the RRAM, during standby mode. The architecture study of set-associative cache made of 6T-2R-2S array is also presented. The 2T-2R-2S/2T-2R mode coupled with SRAM-only mode can be efficiently used to perform CIM for dot product and XNOR computation with co-locating the weights and activations stored onto the same bitcell. System level study highlighting energy efficiency and speedup along with the proposed CIM architecture’s analysis on CIFAR-10 dataset is presented. Design sensitivity analysis with respect to PTM and RRAM parameters is discussed for both the bitcells.

At the intersection of optics and deep learning: statistical inference, computing, and inverse design

Deep learning has been revolutionizing information processing in many fields of science and engineering owing to the massively growing amounts of data and the advances in deep neural network architectures. As these neural networks are expanding their capabilities toward achieving state-of-the-art solutions for demanding statistical inference tasks in various applications, there appears to be a global need for low-power, scalable, and fast computing hardware beyond what existing electronic systems can offer. Optical computing might potentially address some of these needs with its inherent parallelism, power efficiency, and high speed. Recent advances in optical materials, fabrication, and optimization techniques have significantly enriched the design capabilities in optics and photonics, leading to various successful demonstrations of guided-wave and free-space computing hardware for accelerating machine learning tasks using light. In addition to statistical inference and computing, deep learning has also fundamentally affected the field of inverse optical/photonic design. The approximation power of deep neural networks has been utilized to develop optics/photonics systems with unique capabilities, all the way from nanoantenna design to end-to-end optimization of computational imaging and sensing systems. In this review, we attempt to provide a broad overview of the current state of this emerging symbiotic relationship between deep learning and optics/photonics.

Forming‐Free Resistive Switching Memory Crosspoint Arrays for In‐Memory Machine Learning

In‐memory computing (IMC) with crosspoint arrays of resistive switching memory (RRAM) has gained wide attention for accelerating machine learning, data analysis, and deep neural networks. By IMC, matrix‐vector multiplication (MVM) can be executed in the crosspoint array in just one step, thus accelerating a broad range of tasks in machine learning and data analytics. However, a key issue for RRAM crosspoint arrays is the forming operation of the memories which limits the stability and accuracy of the conductance state in the memory device. In this work, a hardware implementation of crosspoint array of forming‐free devices for fast, energy‐efficient accelerators of MVM is reported. RRAM devices with a 1.5 nm‐thick HfO2 layer show an initial low resistance without forming and an analogue‐mode programming behavior for high‐accuracy IMC. Accurate hardware MVM is demonstrated by experimental eigenvalue/eigenvector calculation according to the power‐iteration algorithm, with a fast convergence within about ten iterations to the correct solution. Deflation technique and principal component analysis (PCA) enable the classification of the Iris dataset with 98% accuracy compared with floating‐point implementation. These results support forming‐free crosspoint arrays for accelerating advanced machine learning with IMC.

FPGA Architecture Exploration for DNN Acceleration

Recent years have seen an explosion of machine learning applications implemented on Field-Programmable Gate Arrays (FPGAs) . FPGA vendors and researchers have responded by updating their fabrics to more efficiently implement machine learning accelerators, including innovations such as enhanced Digital Signal Processing (DSP) blocks and hardened systolic arrays. Evaluating architectural proposals is difficult, however, due to the lack of publicly available benchmark circuits. This paper addresses this problem by presenting an open-source benchmark circuit generator that creates realistic DNN-oriented circuits for use in FPGA architecture studies. Unlike previous generators, which create circuits that are agnostic of the underlying FPGA, our circuits explicitly instantiate embedded blocks, allowing for meaningful comparison of recent architectural proposals without the need for a complete inference computer-aided design (CAD) flow. Our circuits are compatible with the VTR CAD suite, allowing for architecture studies that investigate routing congestion and other low-level architectural implications. In addition to addressing the lack of machine learning benchmark circuits, the architecture exploration flow that we propose allows for a more comprehensive evaluation of FPGA architectures than traditional static benchmark suites. We demonstrate this through three case studies which illustrate how realistic benchmark circuits can be generated to target different heterogeneous FPGAs.

A Qualitative Study of the Disorder Effect on the Phonon Transport in a Two-Dimensional Graphene/h-BN Heterostructure

Recently, massive efforts have been made to control phonon transport via introducing disorder. Meanwhile, materials informatics, an advanced material-discovery technology that combines data-driven search algorithms and material property simulations, has made significant progress and shown accurate prediction ability in studying the target properties of new materials. However, with the introduction of disorder, the design space of random structures is greatly expanded. Global optimization for the entire domain is nearly impossible with the current computer resource even when materials informatics reduces the design space to a few percent. Toward the goal of reducing design space, we investigate the effect of different types of disorders on phonon transport in two-dimensional graphene/hexagonal boron nitride heterostructure using non-equilibrium molecular dynamics simulation. The simulation results show that when the hexagonal boron nitride is distributed disorderly in the coherent phonon-dominated structure, that is, the structure with a period length of 1.23 nm, the thermal conductivity is significantly reduced due to the appearance of coherent phonon localization. By qualitatively analyzing different types of disorder, we found that the introduction of disordered structure in the cross direction with a larger shift distance can further reduce the thermal conductivity. Further physical mechanism analysis revealed that the structures with lower thermal conductivity were caused by weak propagation and strong localization of phonon. Our findings have implications for accelerating machine learning in the search for structures with the lowest thermal conductivity, and provide some guidance for the future synthesis of 2D heterostructures with unique thermal properties.

A Golden Decade of Deep Learning: Computing Systems &amp; Applications

Abstract The past decade has seen tremendous progress in the field of artificial intelligence thanks to the resurgence of neural networks through deep learning. This has helped improve the ability for computers to see, hear, and understand the world around them, leading to dramatic advances in the application of AI to many fields of science and other areas of human endeavor. In this essay, I examine the reasons for this progress, including the confluence of progress in computing hardware designed to accelerate machine learning and the emergence of open-source software frameworks to dramatically expand the set of people who can use machine learning effectively. I also present a broad overview of some of the areas in which machine learning has been applied over the past decade. Finally, I sketch out some likely directions from which further progress in artificial intelligence will come.

Photonics-enabled spiking timing-dependent convolutional neural network for real-time image classification.

A photonics-enabled spiking timing-dependent convolutional neural network (CNN) is proposed by manipulating photonics multidimensional parameters in terms of wavelength, temporal and spatial, which breaks the traditional CNN architecture mapping from a spatially parallel to a time-dependent series structure. The proposed CNN with the application of real-time image recognition comprises a photonics convolution processor to accelerate the computing and an involved electronic full connection to execute the classification task. A timing-dependent series of matrix-matrix operations is conducted in the photonics convolution processor that can be achieved based on multidimensional multiplexing by the accumulation of carriers from an active mode-locked laser, dispersion latency induced by a dispersion compensation fiber, and wavelength spatial separation via a waveshaper. Incorporated with the electronic full connection, a photonics-enabled CNN is proven to perform a real-time recognition task on the MNIST database of handwritten digits with a prediction accuracy of 90.04%. Photonics enables conventional neural networks to accelerate machine learning and neuromorphic computing and has the potential to be widely used in information processing and computing, such as goods classification, vowel recognition, and speech identification.

A Software-Circuit-Device Co-Optimization Framework for Neuromorphic Inference Circuits

Neuromorphic circuits, which usually use analog computation for vector-matrix multiplication (VMM) in neural networks (NN), are promising machine learning accelerators with much lower latency and power consumption than digital ones. Analog computation is expected to have a more efficient design space than digital computation since the signals are not digitized. Therefore, it is very suitable for Internet-of-Thing (IoT) applications that require ultra-low power consumption at a low cost. For IoT applications, sometimes it is also desirable to eliminate the digital circuits (such as adders, registers, shifters, multiplexers, and Analog-to-Digital Converters) between the VMM arrays to further reduce the power consumption. However, the optimization of a purely analog circuit is more difficult and requires full SPICE circuit simulations. In this paper, we present a software-circuit-device co-optimization framework using a python wrapper for automatic full circuit SPICE simulation and analysis for neuromorphic circuits. This framework allows users to experiment with how the NN design (software) affects the performance of the hardware neuromorphic circuits. It takes Verilog-A or SPICE models from calibrations or PDK in various technologies and emerging memories (such as ReRAM) without further calibration (unlike using behavior models). We show that the simulation time is reasonable even with hundreds of thousands of synapses under limited computation resources. Using ReRAM and a 45nm generic technology as an example, the effects of feedback network and OpAmp design, software ML architecture, and input data accuracy on the inference accuracy are studied.

Machine-learning-based high-speed lensless large-field holographic projection using double-sampling Fresnel diffraction method

Machine learning can effectively accelerate the runtime of a computer-generated hologram. However, the angular spectrum method and single fast Fresnel transform-based machine learning acceleration algorithms are still limited in the field-of-view angle of projection. In this paper, we propose an efficient method for the fast generation of large field-of-view holograms combining stochastic gradient descent (SGD), neural networks, and double-sampling Fresnel diffraction (DSFD). Compared with the traditional Gerchberg–Saxton (GS) algorithm, the DSFD-SGD algorithm has better reconstruction quality. Our neural network can be automatically trained in an unsupervised manner with a training set of target images without labels, and its combination with the DSFD can improve the optimization speed significantly. The proposed DSFD-Net method can generate 2000-resolution holograms in 0.05 s. The feasibility of the proposed method is demonstrated with simulations and experiments.

End-to-End Synthesis of Dynamically Controlled Machine Learning Accelerators

Edge systems are required to autonomously make real-time decisions based on large quantities of input data under strict power, performance, area, and other constraints. Meeting these constraints is only possible by specializing systems through hardware accelerators purposefully built for machine learning and data analysis algorithms. However, data science evolves at a quick pace, and manual design of custom accelerators has high non-recurrent engineering costs: general solutions are needed to automatically and rapidly transition from the formulation of a new algorithm to the deployment of a dedicated hardware implementation. Our solution is the SOftware Defined Architectures (SODA) Synthesizer, an end-to-end, multi-level, modular, extensible compiler toolchain providing a direct path from machine learning tools to hardware. The SODA Synthesizer frontend is based on the multilevel intermediate representation (MLIR) framework; it ingests pre-trained machine learning models, identifies kernels suited for acceleration, performs high-level optimizations, and prepares them for hardware synthesis. In the backend, SODA leverages state-of-the-art high-level synthesis techniques to generate highly efficient accelerators, targeting both field programmable devices (FPGAs) and application-specific circuits (ASICs). In this paper, we describe how the SODA Synthesizer can also assemble the generated accelerators (based on the finite state machine with datapath model) in a custom system driven by a distributed controller, building a coarse-grained dataflow architecture that does not require a host processor to orchestrate parallel execution of multiple accelerators. We show the effectiveness of our approach by automatically generating ASIC accelerators for layers of popular deep neural networks (DNNs). Our high-level optimizations result in up to 74x speedup on isolated accelerators for individual DNN layers, and our dynamically scheduled architecture yields an additional 3x performance improvement when combining accelerators to handle streaming inputs.

Custom Hardware Inference Accelerator for TensorFlow Lite for Microcontrollers

In recent years, the need for the efficient deployment of Neural Networks (NN) on edge devices has been steadily increasing. However, the high computational demand required for Machine Learning (ML) inference on tiny microcontroller-based IoT devices avoids a direct software deployment on such resource-constrained edge devices. Therefore, various custom and application-specific NN hardware accelerators have been proposed to enable real-time Machine Learning (ML) inference on low-power and resource-limited edge devices. Efficient mapping of the computational load onto hardware and software resources is a key challenge for performance improvement while keeping low power and a low area footprint. High performance and yet low power embedded processors may be attained via the usage of hardware acceleration. This paper presents an efficient hardware-software framework to accelerate machine learning inference on edge devices using a modified TensorFlow Lite for Microcontroller (TFLM) model running on a Microcontroller (MCU) and a dedicated Neural Processing Unit (NPU) custom hardware accelerator, referred to as MCU-NPU. The proposed framework supports weight compression of pruned quantized NN models and exploits the pruned model sparsity to reduce computational complexity further. The proposed methodology has been evaluated by employing the MCU-NPU acceleration for various TFLM-based NN architectures using the common MLPerf Tiny benchmark. Experimental results demonstrate a significant speedup of up to 724x compared to a pure software implementation. For example, the resulting runtime for the CIFAR-10 classification is reduced from about 20 sec to only 37 ms using the proposed hardware acceleration. Moreover, the proposed hardware accelerator outperforms all the reference models optimized for edge devices in terms of inference runtime.

MLoF: Machine Learning Accelerators for the Low-Cost FPGA Platforms

In Internet of Things (IoT) scenarios, it is challenging to deploy Machine Learning (ML) algorithms on low-cost Field Programmable Gate Arrays (FPGAs) in a real-time, cost-efficient, and high-performance way. This paper introduces Machine Learning on FPGA (MLoF), a series of ML IP cores implemented on the low-cost FPGA platforms, aiming at helping more IoT developers to achieve comprehensive performance in various tasks. With Verilog, we deploy and accelerate Artificial Neural Networks (ANNs), Decision Trees (DTs), K-Nearest Neighbors (k-NNs), and Support Vector Machines (SVMs) on 10 different FPGA development boards from seven producers. Additionally, we analyze and evaluate our design with six datasets, and compare the best-performing FPGAs with traditional SoC-based systems including NVIDIA Jetson Nano, Raspberry Pi 3B+, and STM32L476 Nucle. The results show that Lattice’s ICE40UP5 achieves the best overall performance with low power consumption, on which MLoF averagely reduces power by 891% and increases performance by 9 times. Moreover, its cost, power, Latency Production (CPLP) outperforms SoC-based systems by 25 times, which demonstrates the significance of MLoF in endpoint deployment of ML algorithms. Furthermore, we make all of the code open-source in order to promote future research.

Optimization Schemes for In-Memory Linear Regression Circuit With Memristor Arrays

Recently, an in-memory analog circuit based on crosspoint memristor arrays was reported, which enables solving linear regression problems in one step and can be used to train many other machine learning algorithms. To explore its potential for computing accelerator applications, it is of fundamental importance to improve the computing speed of the circuit, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , the circuit response towards correct outputs. In this work, we comprehensively studied the transfer function of this circuit, resulting in a quadratic eigenvalue problem that describes the distribution of poles. The minimal real part of non-zero eigenvalues defines the dominant pole, which in turn dominates the response time. Simulations for multiple linear regression solutions with different datasets evidence that, the computing time does not necessarily increase with problem size. The dominant pole is related to parameters in the circuit, including feedback conductance, and gain bandwidth products of operational amplifiers. By optimizing these parameters synergistically, the dominant pole shifts to higher frequencies and the computing speed is consequently optimized. Our results provide a guideline for design and optimization of in-memory machine learning accelerators with analog memristor arrays. Also, issues including power consumption, impact of noise and variation of sources and memristors are investigated to offer a comprehensive evaluation of the circuit performance.