Learning Accelerators Research Articles

A Novel Low-Power Compression Scheme for Systolic Array-Based Deep Learning Accelerators

The proliferation of deep learning algorithms has catalyzed their utilization to solve a multitude of real-world problems. Algorithms such as deep neural networks (DNNs) are compute- and power-intensive, thereby accentuating the development of hardware platforms like DNN inference accelerators. However, inference execution of large DNNs in resource-constrained environments induces energy bottlenecks in these accelerators. Since large DNNs consist of hundreds of millions of trained parameters, accessing them from the accelerator memory incurs substantial energy. To address this challenge, we propose HardCompress, which, to the best of our knowledge, is the first low-power solution that uses traditional compression strategies pertaining to commercial DNN accelerators in resource-constrained IoT edge devices. The three-step approach involves hardware-based post-quantization trimming of weights, followed by their dictionary-based compression and subsequent decompression by a low-power hardware engine during inference in the accelerator. We evaluate the proposed solution on lightweight networks trained on the MNIST dataset, the compact model trained on the CIFAR-10 dataset, and large DNNs trained on the ImageNet dataset. Performance of HardCompress at different quantization levels has been analyzed. Furthermore, to quantify the effectiveness of the proposed solution, an energy framework that contrasts the DRAM energies of the original and HardCompressed models has been developed. Finally, a fault injection framework which compares the fault resilience of the original model with its HardCompressed counterpart is also proposed. Our results exhibit that HardCompress, without any performance degradation in large DNNs, furnishes a maximum compression of 99.27%, equivalent to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$137\times $ </tex-math></inline-formula> reduction in memory footprint and 0.07 J for 8-bit quantization in the systolic array-based DNN accelerator. Furthermore, our proposed low-power decompression engine incurs an area overhead of only 0.02%; thus, enabling HardCompress’ utilization in resource-constrained environments.

BNN An Ideal Architecture for Acceleration With Resistive in Memory Computation

Binary Neural Networks (BNN) have binarized (-1 and 1) weights and feature maps. Achieving smaller model sizes and computational simplicity, they are well suited for edge-AI systems with power and hardware constraints. Recently, memristive crossbar arrays have gained considerable attention from researchers to perform analog in-memory vector-matrix multiplications in machine learning accelerators, with low power and constant computational time. Crossbar arrays suffer from many non-ideal characteristics such as memristor device imperfections, weight noise, device drift, input/output noises, and DAC/ADC overhead. Thus, for analog AI acceleration to become viable, model architectures must be robust against these non-idealities. We propose that BNN's with their binarized weights, which are ideally mapped to fewer memristive devices with less electrical characteristic issues and higher tolerance to computational noise, are a promising architecture for analog computation. In this work, we examine the viability of deploying state of the art BNNs, with features such as real value residual connections and parametric activations with biases, to analog in-memory computational accelerators. Our simulations show that BNNs are significantly more robust to crossbar non-idealities than full-precision networks, require less chip area, and consume less power on memristive crossbar architectures.

The impact of ICT characteristics on students’ learning outcomes during COVID-19 lockdown period: The crucial role of techno eustress perception

BACKGROUND: The COVID-19 crisis led to an unprecedented acceleration of digital learning. It pushed many institutions to abruptly switch to fully online learning modes from face-to-face learning. Prior studies show that higher IT demands can cause challenge or hindrance stressors, depending on how the digital technology characteristics are perceived by the end-user. However, there is a gap in our knowledge regarding how ICT characteristics can lead to positive stress appraisals in a remote learning environment. OBJECTIVE: This paper leverages the person-environment fit and technostress literature to examine how usefulness and reliability as demand-ability stressors of ICT tools can positively impact learning outcomes among remote learning students. Techno eustress perceptions are evaluated as a crucial mechanism for theorizing the positive impact. METHODS: We used the survey method, sampling students (N = 82) during the lockdown period to test this model. RESULTS: Our findings highlight the ICT characteristic of usefulness as salient in contributing to student learning outcomes as it promotes techno eustress. CONCLUSIONS: This is the first study to demonstrate a positive impact of ICT characteristics on student learning outcomes via techno eustress perceptions.

AroMa : Evaluating Deep Learning Systems for Stealthy Integrity Attacks on Multi-tenant Accelerators

Multi-tenant applications have been proliferating in recent years, supported by the emergence of computing-as-service paradigms. Unfortunately, multi-tenancy induces new security vulnerabilities due to spatial or temporal co-location of applications with possibly malicious intent. In this article, we consider a special class of stealthy integrity attacks on multi-tenant deep learning accelerators. One interesting conclusion is that it is possible to perform targeted integrity attacks on kernel weights of deep learning systems such that it remains functional but mis-labels specific categories of input data through standard RowHammer attacks by only changing 0.0009% of the total weights. We develop an automated framework, AroMa , to evaluate the impact of multi-tenancy on security of deep learning accelerators against integrity attacks on memory systems. We present extensive evaluations on AroMa to demonstrate its effectiveness.

FPGA Implementation of a Deep Learning Acceleration Core Architecture for Image Target Detection

Due to the flexibility and ease of deployment of Field Programmable Gate Arrays (FPGA), more and more studies have been conducted on developing and optimizing target detection algorithms based on Convolutional Neural Networks (CNN) models using FPGAs. Still, these studies focus on improving the performance of the core algorithm and optimizing hardware structure, with few studies focusing on the unified architecture design and corresponding optimization techniques for the algorithm model, resulting in inefficient overall model performance. The essential reason is that these studies do not address arithmetic power, speed, and resource consistency. In order to solve this problem, we propose a deep learning acceleration core architecture based on FPGAs, which is designed for target detection algorithms with CNN models, using multi-channel parallelization of CNN network models to improve the arithmetic power, using scheduling tasks and intensive computation pipelining to meet the algorithm’s data bandwidth requirements and unifying the speed and area of the orchestrated computation matrix to save hardware resources. The proposed framework achieves 14 Frames Per Second (FPS) inference performance of the TinyYolo model at 5 Giga Operations Per Second (GOPS) with 30% higher running clock frequency, 2–4 times higher arithmetic power, and 28% higher Digital Signal Processing (DSP) resource utilization efficiency using less than 25% of FPGA resource usage.

A 1.6-mW Sparse Deep Learning Accelerator for Speech Separation

Low-power deep learning accelerators (DLAs) on the speech processing enable real-time applications on edge devices. However, most of the existing accelerators suffer from high-power consumption and focus on image applications only. This article presents a low-power accelerator for speech separation through algorithm and hardware optimizations. At the algorithm level, the model is compressed with structured sensitivity as well as unstructured pruning, and further quantized to the shifted 8-bit floating-point format instead of the 32-bit floating-point format. The computations with the zero kernel and zero activation values are skipped by decomposition of the dilated and transposed convolutions. At the hardware level, the compressed model is then supported by an architecture with eight independent multipliers and accumulators (MACs) with a simple zero-skipping hardware to take advantage of the activation sparsity and low-power processing. The proposed approach reduces the model size by 95.44% and computation complexity by 93.88%. The final implementation with the TSMC 40-nm process can achieve real-time speech separation and consumes 1.6-mW power when operated at 150 MHz. The normalized energy efficiency and area efficiency are 2.344 TOPS/W and 14.42 GOPS/mm2, respectively.

Device Modeling Bias in ReRAM-Based Neural Network Simulations

Emerging technologies based on resistive switching (ReRAM) devices promise to improve the speed and energy efficiency of next generation machine learning accelerators, but further research is required for achieving commercial maturity. System-level prototyping with emerging devices is costly, and algorithmic investigations require hardware neural network modeling which often deviates from experimental reality. In this work, the concept of modeling bias is proposed as a way to quantify this deviation and support reliable evaluation of device populations in the context of neural network algorithms. While applicable to other device modeling techniques, modeling bias is investigated here using jump tables - a promising physics-less technique to model emerging memory devices for hardware networks. Questions about the fidelity of these tables in relation to stochastic device behavior are answered. Two methods of jump table modeling – binning and a novel Optuna-optimized binning - are explored using synthetic data with known distributions for benchmarking and experimental data obtained from TiOx ReRAM devices for practical testing. Novel device metrics are proposed, and it is shown that these metrics can present crucial insights on the device population prior to training the hardware network. Results on a multi-layer perceptron trained on MNIST show that device models based on binning deviate from target network accuracy at a low number of points and high switching noise in the device dataset. The proposed approach opens the possibility for device-algorithm co-design investigations into statistical device models with better performance, as well as experimentally verified modeling bias in different in-memory computing and neural network architectures.

Robust Architecture-Agnostic and Noise Resilient Training of Photonic Deep Learning Models

Neuromorphic photonic accelerators for Deep Learning (DL) have increasingly gained attention over the recent years due to their ability for ultra fast matrix-based calculations and low power consumption providing a great potential for DL implementations to deal with a wide range of different applications. At the same time, physical properties of the optical components hinder their application since they introduce a number of limitations, such as easily saturated activation functions as well as the existence of various noise sources. As a result, photonic DL models are especially challenging to be trained and deployed, compared with regular DL models, since traditionally used methods do not take into account the aforementioned constraints. To overcome these limitations and motivated by the fact that the information lost in one layer cannot be easily recovered when gradient-descent based algorithms are employed, we propose a novel training method for photonic neuromorphic architectures that is capable of taking into account a wide range of limitations of the actual hardware, including noise sources and easily saturated activation mechanisms. Compared to existing works, the proposed method takes a more holistic view of the training process, focusing both on the initialization process, as well as on the actual weight updates. The effectiveness of the proposed method is demonstrated on a variety of different problems and photonic neural network (PNN) architectures, including a noisy photonic recurrent neural network evaluated on high-frequency time series forecasting and a deep photonic feed-forward setup consisting of a transmitter, noisy channel, and receiver, which is used as an intensity modulation/direct detection system (IM/DD).

On-Line Fault Protection for ReRAM-Based Neural Networks

The emerging Resistive RAM (ReRAM) technology significantly boosts the performance and the energy efficiency of the deep learning accelerators (DLAs) via the Computing-in-Memory (CiM) architecture. However, ReRAM-based DLA also suffers a high occurrence rate of memory faults. How to detect and protect against the faults in ReRAM devices poses great challenges to ReRAM-based DLA design. In this work, we propose RRAMedy, an in-situ fault detection and network remedy framework for ReRAM-based DLAs. With the proposed Adversarial Example Testing, which is a lifetime on-device and on-line fault detection technique, it achieves high detection coverage of both hard faults and soft faults at a low run-time cost. In addition, it employs an edge-cloud collaborative model retraining method to tolerate the detected faults by leveraging the inherent fault-adaptive capability of DNNs. Meanwhile, to enable in-situ model remedy when the cloud assistance is absent due to security or overhead issues, we propose to accelerate the fault-masking retraining process on edge devices with parallelized Knowledge Transfer. Our experimental results show that the proposed fault detection technique achieves high fault detection accuracy and delivers real-time testing performance. Meanwhile, the proposed retraining approach greatly alleviates the accuracy degradation problem and achieves excellent performance speedups over the baselines.

Lightweight and Energy-Efficient Deep Learning Accelerator for Real-Time Object Detection on Edge Devices.

Tiny machine learning (TinyML) has become an emerging field according to the rapid growth in the area of the internet of things (IoT). However, most deep learning algorithms are too complex, require a lot of memory to store data, and consume an enormous amount of energy for calculation/data movement; therefore, the algorithms are not suitable for IoT devices such as various sensors and imaging systems. Furthermore, typical hardware accelerators cannot be embedded in these resource-constrained edge devices, and they are difficult to drive real-time inference processing as well. To perform the real-time processing on these battery-operated devices, deep learning models should be compact and hardware-optimized, and hardware accelerator designs also have to be lightweight and consume extremely low energy. Therefore, we present an optimized network model through model simplification and compression for the hardware to be implemented, and propose a hardware architecture for a lightweight and energy-efficient deep learning accelerator. The experimental results demonstrate that our optimized model successfully performs object detection, and the proposed hardware design achieves 1.25× and 4.27× smaller logic and BRAM size, respectively, and its energy consumption is approximately 10.37× lower than previous similar works with 43.95 fps as a real-time process under an operating frequency of 100 MHz on a Xilinx ZC702 FPGA.

A Survey on Optical Phase-Change Memory: The Promise and Challenges

Silicon photonics (SiPh) technology has facilitated the deployment of integrated photonics across different application domains, from ultra-fast communication in Datacom applications to energy-efficient optical computation in emerging hardware accelerators for machine learning. More recently, the integration of SiPh and phase change materials has created a unique opportunity to realize adaptable, reconfigurable, and programmable photonic platforms. In particular, the nonvolatile programmability in phase change materials has made them a promising candidate for implementing photonic memory cells and architectures. Accordingly, photonic memory systems and even in-memory photonic computing paradigms are on the rise, especially given their potential for improving data access in electronic and photonic processors. However, there are still many challenges in the design and fabrication of phase-change photonic integrated circuits, which need to be addressed. This article presents a comprehensive survey on the recent advances and challenges for the integration of phase change materials with contemporary photonic devices while focusing on the photonic memory application. In particular, we explore phase-change photonic memory from the material level to the architecture level by presenting an overview of different material-level characteristics of phase change materials with their optical, electrical, and thermal properties as well as their integration into SiPh devices and photonic memory architectures and their application for in-memory photonic computing. We also present a comparison with electronic memory and discuss open research challenges that must be addressed to further advance phase-change photonic memory towards successful integration into emerging computing systems.

FARANE-Q: Fast Parallel and Pipeline Q-Learning Accelerator for Configurable Reinforcement Learning SoC

This paper proposes a FAst paRAllel and pipeliNE Q-learning accelerator (FARANE-Q) for a configurable Reinforcement Learning (RL) algorithm implemented in a System on Chip (SoC). The proposed work offers flexibility, configurability, and scalability while maintaining computation speed and accuracy to overcome the challenges of a dynamic environment and increasing complexity. The proposed method includes a Hardware/Software (HW/SW) design methodology for the SoC architecture to achieve flexibility. We also propose joint optimizations on the algorithm, architecture, and implementation to obtain optimum (high efficiency) performance, specifically in energy and area efficiency. Furthermore, we implemented the proposed design in a real-time Zynq Ultra96-V2 FPGA platform to evaluate the functionality with an actual use case of smart navigation. Experimental results confirm that the proposed accelerator FARANE-Q outperforms state-of-the-art works by achieving a throughput of up to 148.55 MSps. It corresponds to the energy efficiency of 1747.64 MSps/W per agent for 32-bit and 2424.33 MSps/W per agent for 16-bit FARANE-Q. Moreover, the proposed 16-bit FARANE-Q outperforms other related works by an improvement of at least <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.23\times $ </tex-math></inline-formula> in energy efficiency. The designed system also maintains an error accuracy of less than 0.4% with optimized bit precision for more than eight fraction bits. The proposed FARANE-Q also offers a speed up of processing time up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1795\times $ </tex-math></inline-formula> compared to embedded SW computation executed on ARM Zynq processor and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$280\times $ </tex-math></inline-formula> of computation of full software executed on i7 processor. Hence, the proposed work has the potential to be used for smart navigation, robotic control, and predictive maintenance.

A Novel Mixed Precision Distributed TPU GAN for Accelerated Learning Curve

Deep neural networks are gaining importance and popularity in applications and services. Due to the enormous number of learnable parameters and datasets, the training of neural networks is computationally costly. Parallel and distributed computation-based strategies are used to accelerate this training process. Generative Adversarial Networks (GAN) are a recent technological achievement in deep learning. These generative models are computationally expensive because a GAN consists of two neural networks and trains on enormous datasets. Typically, a GAN is trained on a single server. Conventional deep learning accelerator designs are challenged by the unique properties of GAN, like the enormous computation stages with non-traditional convolution layers. This work addresses the issue of distributing GANs so that they can train on datasets distributed over many TPUs (Tensor Processing Unit). Distributed learning training accelerates the learning process and decreases computation time. In this paper, the Generative Adversarial Network is accelerated using the distributed multi-core TPU in distributed data-parallel synchronous model. For adequate acceleration of the GAN network, the data parallel SGD (Stochastic Gradient Descent) model is implemented in multi-core TPU using distributed TensorFlow with mixed precision, bfloat16, and XLA (Accelerated Linear Algebra). The study was conducted on the MNIST dataset for varying batch sizes from 64 to 512 for 30 epochs in distributed SGD in TPU v3 with 128 × 128 systolic array. An extensive batch technique is implemented in bfloat16 to decrease the storage cost and speed up floating-point computations. The accelerated learning curve for the generator and discriminator network is obtained. The training time was reduced by 79% by varying the batch size from 64 to 512 in multi-core TPU.

Recent IEEE SSCS Silicon Valley Chapter News: Seminars, Workshops, Distinguished Lectures, and Silicon Valley Engineering Hall of Fame Awards Ceremony Honoring Dr. Bill Dally, Nvidia and Stanford [Chapters

The IEEE Solid-State Circuits Society (SSCS) Santa Clara Valley (SCV) Chapter held several well-attended seminars, webinars, workshops, and Distinguished Lecturer (DL) talks over the past three years, throughout the pandemic, and recently attended the Silicon Valley Engineering Council (SVEC) Engineers Week Banquet and Hall of Fame Awards Ceremony, in February 2023, honoring Dr. Bill Dally, chief scientist, Nvidia, and adjunct professor, Stanford University, who is also well known in our SSCS community. This article highlights the wide variety of events covering topics from mixed-signal and analog circuits, high-speed links, machine learning acceleration, and the semiconductor industry. The speakers included professors and postdoctoral scholars from academia as well as start-up founders, technology leaders, and executives from industry. Please contact us if you are interested in presenting at our meetings, would like to attend Chapter events, or wish to volunteer. Full information on Chapter activities and upcoming seminars is available on the Chapter website: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://r6.ieee.org/scv-sscs/</uri> .

In-Memory Computing for Machine Learning and Deep Learning

In-memory computing (IMC) aims at executing numerical operations via physical processes, such as current summation and charge collection, thus accelerating common computing tasks including the matrix-vector multiplication. While extremely promising for memory-intensive processing such as machine learning and deep learning, the IMC design and realization must face significant challenges due to device and circuit nonidealities. This work provides an overview of the research trends and options for IMC-based implementations of deep learning accelerators with emerging memory technologies. The device technologies, the computing primitives, and the digital/analog/mixed design approaches are presented. Finally, the major device issues and metrics for IMC are discussed and benchmarked.

DeepEdgeSoC: End-to-end deep learning framework for edge IoT devices

The acceleration of deep neural networks (DNNs) on edge devices is gaining significant importance in various application domains. General purpose graphics processing units (GPGPUs) are typically used to explore, train and evaluate DNNs because they offer higher processing and computational capability compared to CPUs. However, this comes at the cost of increased power consumption required by these devices for operation, which prevents efficient deployment of networks on edge devices. In the Internet of Things (IoT) domain, Field programmable gate arrays (FPGAs) are considered a powerful alternative since their flexible architecture can run the DNNs with much less energy. The enormous amount of effort and time required for the entire end-to-end edge-aware deployment urged us to develop DeepEdgeSoc, an integrated framework for deep learning (DL) design and acceleration. DeepEdgeSoc is an overarching framework under which DNNs can be built. DeepGUI, a visual drag-and-drop DNN design environment, plays an important role in accelerating the network design phase. In DeepEdgeSoc, the networks can be quantized and compressed to suite the underlying edge devices in terms of size and energy. DeepEdgeSoc goes beyond the software level by converting the networks to appropriate FPGA implementations that can be directly synthesized and integrated within a System-on-Chip (SoC).

A Nonvolatile All-Spin Nonbinary Matrix Multiplier: An Efficient Hardware Accelerator for Machine Learning

We propose and analyze a compact and non-volatile nanomagnetic (all-spin) non-binary matrix multiplier performing the multiply-and-accumulate (MAC) operation using two magnetic tunnel junctions - one activated by strain to act as the multiplier, and the other activated by spin-orbit torque pulses to act as a domain wall synapse that performs the operation of the accumulator. It has two advantages over the usual crossbar-based electronic non-binary matrix multiplier. First, while the crossbar architecture requires N3 devices to multiply two matrices, we require only 2N2 devices. Second, our matrix multiplier is non-volatile and retains the information about the product matrix after being powered off. Here, we present an example where each MAC operation can be performed in ~5 ns and the maximum energy dissipated per operation is ~60Nmax aJ, where Nmax is the largest matrix size. This provides a very useful hardware accelerator for machine learning and artificial intelligence tasks which involve the multiplication of large matrices. The non-volatility allows the matrix multiplier to be embedded in powerful non-von-Neumann architectures, including processor-in-memory. It also allows much of the computing to be done at the edge (of internet-of-things) while reducing the need to access the cloud, thereby making artificial intelligence more resilient against cyberattacks.

Guest Editorial: IEEE TC Special Issue: Hardware Acceleration of Machine Learning

The papers in this special section explore research on the latest topics related to the hardware acceleration of machine learning.

DLA-H: A Deep Learning Accelerator for Histopathologic Image Classification.

It is more than a decade since machine learning and especially its leading subtype deep learning have become one of the most interesting topics in almost all areas of science and industry. In numerous contexts, at least one of the applications of deep learning is utilized or is going to be utilized. Using deep learning for image classification is now very popular and widely used in various use cases. Many types of research in medical sciences have been focused on the advantages of deep learning for image classification problems. Some recent researches show more than 90% accuracy for breast tissue classification which is a breakthrough. A huge number of computations in deep neural networks are considered a big challenge both from software and hardware point of view. From the architectural perspective, this big amount of computing operations will result in high power consumption and computation runtime. This led to the emersion of deep learning accelerators which are designed mainly for improving performance and energy efficiency. Data reuse and localization are two great opportunities for achieving energy-efficient computations with lower runtime. Data flows are mainly designed based on these important parameters. In this paper, DLA-H and BJS, a deep learning accelerator, and its data flow for histopathologic image classification are proposed. The simulation results with the MAESTRO tool showed 756 cycles for total runtime and [Formula: see text] GFLOPS roofline throughput that is an extreme performance improvement in comparison to current general-purpose deep learning accelerators and data flows.

Architecting Decentralization and Customizability in DNN Accelerators for Hardware Defect Adaptation

The efficiency of machine intelligence techniques has improved noticeably in the embedded application domains thanks to the dedicated hardware accelerators for deep neural networks (DNNs). Despite the economic criticality of yield and reliability problems in advanced semiconductor nodes, these concerns have attracted limited attention in the context of embedded machine intelligence devices. The micro-architectural features of deep learning accelerators, when paired with the algorithmic characteristics of DNNs, unlock novel opportunities to tackle semiconductor reliability problems in embedded deep learning devices. While the fine-grained bypassing of the faulty processing elements reins the computational impact of hardware defects, a one-time training of DNNs with Hardware-Aware Dropout/Dropconnect techniques boosts model decentralization and facilitates accurate neural network inference in the degraded computational fabrics. Furthermore, on-device calibration methods can improve resilience even further without necessitating expensive defect compensation methods such as device-specific training. Our work confirms the potential for improving the yield, reliability, and operational lifetime of embedded machine intelligence devices through a highly practical co-design of DNNs and configurable hardware architectures.