Deep Learning Accelerators Research Articles

A Novel Mixed Precision Distributed TPU GAN for Accelerated Learning Curve

Deep neural networks are gaining importance and popularity in applications and services. Due to the enormous number of learnable parameters and datasets, the training of neural networks is computationally costly. Parallel and distributed computation-based strategies are used to accelerate this training process. Generative Adversarial Networks (GAN) are a recent technological achievement in deep learning. These generative models are computationally expensive because a GAN consists of two neural networks and trains on enormous datasets. Typically, a GAN is trained on a single server. Conventional deep learning accelerator designs are challenged by the unique properties of GAN, like the enormous computation stages with non-traditional convolution layers. This work addresses the issue of distributing GANs so that they can train on datasets distributed over many TPUs (Tensor Processing Unit). Distributed learning training accelerates the learning process and decreases computation time. In this paper, the Generative Adversarial Network is accelerated using the distributed multi-core TPU in distributed data-parallel synchronous model. For adequate acceleration of the GAN network, the data parallel SGD (Stochastic Gradient Descent) model is implemented in multi-core TPU using distributed TensorFlow with mixed precision, bfloat16, and XLA (Accelerated Linear Algebra). The study was conducted on the MNIST dataset for varying batch sizes from 64 to 512 for 30 epochs in distributed SGD in TPU v3 with 128 × 128 systolic array. An extensive batch technique is implemented in bfloat16 to decrease the storage cost and speed up floating-point computations. The accelerated learning curve for the generator and discriminator network is obtained. The training time was reduced by 79% by varying the batch size from 64 to 512 in multi-core TPU.

In-Memory Computing for Machine Learning and Deep Learning

In-memory computing (IMC) aims at executing numerical operations via physical processes, such as current summation and charge collection, thus accelerating common computing tasks including the matrix-vector multiplication. While extremely promising for memory-intensive processing such as machine learning and deep learning, the IMC design and realization must face significant challenges due to device and circuit nonidealities. This work provides an overview of the research trends and options for IMC-based implementations of deep learning accelerators with emerging memory technologies. The device technologies, the computing primitives, and the digital/analog/mixed design approaches are presented. Finally, the major device issues and metrics for IMC are discussed and benchmarked.

DeepEdgeSoC: End-to-end deep learning framework for edge IoT devices

The acceleration of deep neural networks (DNNs) on edge devices is gaining significant importance in various application domains. General purpose graphics processing units (GPGPUs) are typically used to explore, train and evaluate DNNs because they offer higher processing and computational capability compared to CPUs. However, this comes at the cost of increased power consumption required by these devices for operation, which prevents efficient deployment of networks on edge devices. In the Internet of Things (IoT) domain, Field programmable gate arrays (FPGAs) are considered a powerful alternative since their flexible architecture can run the DNNs with much less energy. The enormous amount of effort and time required for the entire end-to-end edge-aware deployment urged us to develop DeepEdgeSoc, an integrated framework for deep learning (DL) design and acceleration. DeepEdgeSoc is an overarching framework under which DNNs can be built. DeepGUI, a visual drag-and-drop DNN design environment, plays an important role in accelerating the network design phase. In DeepEdgeSoc, the networks can be quantized and compressed to suite the underlying edge devices in terms of size and energy. DeepEdgeSoc goes beyond the software level by converting the networks to appropriate FPGA implementations that can be directly synthesized and integrated within a System-on-Chip (SoC).

FPGA-based Acceleration of Time Series Similarity Prediction: From Cloud to Edge

With the proliferation of low-cost sensors and the Internet of Things, the rate of producing data far exceeds the compute and storage capabilities of today’s infrastructure. Much of this data takes the form of time series, and in response, there has been increasing interest in the creation of time series archives in the past decade, along with the development and deployment of novel analysis methods to process the data. The general strategy has been to apply a plurality of similarity search mechanisms to various subsets and subsequences of time series data to identify repeated patterns and anomalies; however, the computational demands of these approaches renders them incompatible with today’s power-constrained embedded CPUs. To address this challenge, we present FA-LAMP, an FPGA-accelerated implementation of the Learned Approximate Matrix Profile (LAMP) algorithm, which predicts the correlation between streaming data sampled in real-time and a representative time series dataset used for training. FA-LAMP lends itself as a real-time solution for time series analysis problems such as classification. We present the implementation of FA-LAMP on both edge- and cloud-based prototypes. On the edge devices, FA-LAMP integrates accelerated computation as close as possible to IoT sensors, thereby eliminating the need to transmit and store data in the cloud for posterior analysis. On the cloud-based accelerators, FA-LAMP can execute multiple LAMP models on the same board, allowing simultaneous processing of incoming data from multiple data sources across a network. LAMP employs a Convolutional Neural Network (CNN) for prediction. This work investigates the challenges and limitations of deploying CNNs on FPGAs using the Xilinx Deep Learning Processor Unit (DPU) and the Vitis AI development environment. We expose several technical limitations of the DPU, while providing a mechanism to overcome them by attaching custom IP block accelerators to the architecture. We evaluate FA-LAMP using a low-cost Xilinx Ultra96-V2 FPGA as well as a cloud-based Xilinx Alveo U280 accelerator card and measure their performance against a prototypical LAMP deployment running on a Raspberry Pi 3, an Edge TPU, a GPU, a desktop CPU, and a server-class CPU. In the edge scenario, the Ultra96-V2 FPGA improved performance and energy consumption compared to the Raspberry Pi; in the cloud scenario, the server CPU and GPU outperformed the Alveo U280 accelerator card, while the desktop CPU achieved comparable performance; however, the Alveo card offered an order of magnitude lower energy consumption compared to the other four platforms. Our implementation is publicly available at https://github.com/aminiok1/lamp-alveo.

DLA-H: A Deep Learning Accelerator for Histopathologic Image Classification.

It is more than a decade since machine learning and especially its leading subtype deep learning have become one of the most interesting topics in almost all areas of science and industry. In numerous contexts, at least one of the applications of deep learning is utilized or is going to be utilized. Using deep learning for image classification is now very popular and widely used in various use cases. Many types of research in medical sciences have been focused on the advantages of deep learning for image classification problems. Some recent researches show more than 90% accuracy for breast tissue classification which is a breakthrough. A huge number of computations in deep neural networks are considered a big challenge both from software and hardware point of view. From the architectural perspective, this big amount of computing operations will result in high power consumption and computation runtime. This led to the emersion of deep learning accelerators which are designed mainly for improving performance and energy efficiency. Data reuse and localization are two great opportunities for achieving energy-efficient computations with lower runtime. Data flows are mainly designed based on these important parameters. In this paper, DLA-H and BJS, a deep learning accelerator, and its data flow for histopathologic image classification are proposed. The simulation results with the MAESTRO tool showed 756 cycles for total runtime and [Formula: see text] GFLOPS roofline throughput that is an extreme performance improvement in comparison to current general-purpose deep learning accelerators and data flows.

Benchmarking edge computing devices for grape bunches and trunks detection using accelerated object detection single shot multibox deep learning models

Purpose:Visual perception enables robots to perceive the environment. Visual data is processed using computer vision algorithms that are usually time-expensive and require powerful devices to process the visual data in real-time, which is unfeasible for open-field robots with limited energy. This work benchmarks the performance of different heterogeneous platforms for object detection in real-time. This research benchmarks three architectures: embedded GPU—Graphical Processing Units (such as NVIDIA Jetson Nano 2GB and 4GB, and NVIDIA Jetson TX2), TPU—Tensor Processing Unit (such as Coral Dev Board TPU), and DPU—Deep Learning Processor Unit (such as in AMD/Xilinx ZCU104 Development Board, and AMD/Xilinx Kria KV260 Starter Kit). Methods:The authors used the RetinaNet ResNet-50 fine-tuned using the natural VineSet dataset. After the trained model was converted and compiled for target-specific hardware formats to improve the execution efficiency. Conclusions and Results:The platforms were assessed in terms of performance of the evaluation metrics and efficiency (time of inference). Graphical Processing Units (GPUs) were the slowest devices, running at 3FPS to 5FPS, and Field Programmable Gate Arrays (FPGAs) were the fastest devices, running at 14FPS to 25FPS. The efficiency of the Tensor Processing Unit (TPU) is irrelevant and similar to NVIDIA Jetson TX2. TPU and GPU are the most power-efficient, consuming about 5W. The performance differences, in the evaluation metrics, across devices are irrelevant and have an F1 of about 70% and mean Average Precision (mAP) of about 60%.

Feasibility Analysis and Implementation of Adaptive Dynamic Reconfiguration of CNN Accelerators

In multi-tasking scenarios with dynamically changing loads, the parallel computing of convolutional neural networks (CNNs) causes high energy and resource consumption in the system. Another critical problem is that previous neural network hardware accelerators are often limited to fixed scenarios and lack the function of adaptive adjustment. To solve these problems, a reconfiguration adaptive system based on the prediction of algorithm workload is proposed in this paper. Deep Learning Processor Unit (DPU) from Xilinx has excellent performance in accelerating network computing. After summarizing the characteristics of hardware accelerators and gaining an in-depth understanding of the DPU structure, we propose a regression model for CNNs runtime prediction and a guidance scheme for adaptive reconfiguration combined with the characteristics of Deep Learning Processor Unit. For different DPU sizes, the accuracy of the proposed prediction model achieves 90.7%. With the dynamic reconfiguration technology, the proposed strategy can enable accurate and fast reconfiguration. In the load change scenario, the proposed system can significantly reduce power consumption.

Compute-in-Memory Technologies and Architectures for Deep Learning Workloads

The use of deep learning (DL) to real-world applications, such as computer vision, speech recognition, and robotics, has become ubiquitous. This can be largely attributed to a virtuous cycle between algorithms, data and computing, and storage capacity, which has driven rapid advances in all these dimensions. The ever-increasing demand for computation and memory from DL workloads presents challenges across the entire spectrum of computing platforms, from edge devices to the cloud. Hence, there is a need to explore new hardware paradigms that go well beyond the current mainstays such as graphical processing units (GPUs), tensor processing units (TPUs), and neural processing units (NPUs). A key bottleneck of current platforms is the so-called memory wall, which arises from the need to move large amounts of data between memory and compute units, expending considerable time and energy. One promising solution to this challenge is to move some computations either closer to memory, within the memory subsystem, or even within individual memory arrays. This approach, which is broadly referred to as compute-in-memory (CiM), has the potential to break the memory wall, and thereby greatly improve speed and power consumption. In this article, we provide an overview of CiM techniques used at different levels of the memory hierarchy and based on different memory technologies, including static random access memories (SRAMs), nonvolatile memories (NVMs), and DRAMs. We also discuss architectural approaches to designing CiM-based DL accelerators. Finally, we discuss the challenges associated with adopting CiM in future DL accelerators.

AIDA: Associative In-Memory Deep Learning Accelerator

This work presents an associative in-memory deep learning processor (AIDA) for edge devices. An associative processor is a massively parallel non-von Neumann accelerator that uses memory cells for computing; the bulk of data is never transferred outside the memory arrays for external processing. AIDA utilizes a dynamic content addressable memory for both data storage and processing, and benefits from sparsity and limited arithmetic precision, typical in modern deep neural networks. The novel in-data processing implementation designed for the AIDA accelerator achieves a speedup of 270× over an advanced central processing unit at more than three orders-of-magnitude better energy efficiency.

Architecting Decentralization and Customizability in DNN Accelerators for Hardware Defect Adaptation

The efficiency of machine intelligence techniques has improved noticeably in the embedded application domains thanks to the dedicated hardware accelerators for deep neural networks (DNNs). Despite the economic criticality of yield and reliability problems in advanced semiconductor nodes, these concerns have attracted limited attention in the context of embedded machine intelligence devices. The micro-architectural features of deep learning accelerators, when paired with the algorithmic characteristics of DNNs, unlock novel opportunities to tackle semiconductor reliability problems in embedded deep learning devices. While the fine-grained bypassing of the faulty processing elements reins the computational impact of hardware defects, a one-time training of DNNs with Hardware-Aware Dropout/Dropconnect techniques boosts model decentralization and facilitates accurate neural network inference in the degraded computational fabrics. Furthermore, on-device calibration methods can improve resilience even further without necessitating expensive defect compensation methods such as device-specific training. Our work confirms the potential for improving the yield, reliability, and operational lifetime of embedded machine intelligence devices through a highly practical co-design of DNNs and configurable hardware architectures.

OnSRAM: Efficient Inter-Node On-Chip Scratchpad Management in Deep Learning Accelerators

Hardware acceleration of Artificial Intelligence (AI) workloads has gained widespread popularity with its potential to deliver unprecedented performance and efficiency. An important challenge remains in how AI accelerators are programmed to sustain high utilization without impacting end-user productivity. Prior software optimizations start with an input graph and focus on node-level optimizations, viz. dataflows and hierarchical tiling, and graph-level optimizations such as operation fusion. However, little effort has been devoted to inter-node on-chip scratchpad memory (SPM) management in Deep Learning (DL) accelerators, whose significance is bolstered by the recent trends in complex network topologies and the emergence of eager execution in DL frameworks. We characterize and show that there exists up to a 5.2× performance gap in DL inference to be bridged using SPM management and propose OnSRAM, a novel SPM management framework integrated with the compiler runtime of a DL accelerator. We develop two variants, viz. OnSRAM-Static, which works on static graphs to identify data structures that can be lucratively held on-chip based on their size, liveness and significance, and OnSRAM-Eager, which targets an eager execution model (no graph) and uses a history-based speculative scheme to hold/discard data structures. We integrate OnSRAM with TensorFlow and analyze it on multiple accelerator configurations. Across a suite of 12 images, objects, and language networks, on a 3 TFLOP system with a 2 MB SPM and 32 GBps external memory bandwidth, OnSRAM-Static and OnSRAM-Eager achieve 1.02–4.8× and 1.02–3.1× reduction in inference latency (batch size of 1), over a baseline with no SPM management. In terms of energy savings, we observe average reductions of 1.51× (up to 4.1×) and 1.23× (up to 2.9×) for the static and eager execution scenarios, respectively.

HyCA: A Hybrid Computing Architecture for Fault-Tolerant Deep Learning

Hardware faults on the regular 2-D computing array of a typical deep learning accelerator (DLA) can lead to dramatic prediction accuracy loss. Prior redundancy design approaches typically have each homogeneous redundant processing element (PE) to mitigate faulty PEs for a limited region of the 2-D computing array rather than the entire computing array to avoid the excessive hardware overhead. However, they fail to recover the computing array when the number of faulty PEs in any region exceeds the number of redundant PEs in the same region. The mismatch problem deteriorates when the fault injection rate rises and the faults are unevenly distributed. To address the problem, we propose a hybrid computing architecture (HyCA) for fault-tolerant DLAs. It has a set of dot-production processing units (DPPUs) to recompute all the operations that are mapped to the faulty PEs despite the faulty PE locations. According to our experiments, HyCA shows significantly higher reliability, scalability, and performance with less chip area penalty when compared to the conventional redundancy approaches. Moreover, by taking advantage of the flexible recomputing, HyCA can also be utilized to scan the entire 2-D computing array and detect the faulty PEs effectively at runtime.

An 8.9–71.3 TOPS/W Deep Learning Accelerator for Arbitrarily Quantized Neural Networks

This brief presents the first general-purpose deep learning accelerator for arbitrarily quantized neural networks. A pre-processing and corresponding multiply-and-accumulate operator are proposed to perform the vector products between arbitrarily quantized two vectors. As a result, the accelerator can support precision scalability of quantization up to 4b for one of input activation and weight, and up to 8b for the other, respectively. The proposed approach shows a reduction of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$0.57\times $ </tex-math></inline-formula> area and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$0.78\times $ </tex-math></inline-formula> power consumption, respectively, compared to those with the conventional 8b linear precision which is required to perform an equal level of task accuracy. The implemented accelerator in 28nm CMOS operates at an operating frequency range of 100-to-250MHz over a supply voltage range of 0.75-to-1.2V. Depending on configurations, it shows an energy efficiency of 8.9-to-71.3TOPS/W.

Topologically Protected All‐Optical Memory (Adv. Electron. Mater. 10/2022)

All-Optical Memories Realizing photonic in-memory processors is critical for light-based deep learning accelerators. In article number 2200579 Namkyoo Park, Sunkyu Yu, and co-workers demonstrate an all-optical building block for in-memory processors by exploiting topological states of dynamical parity-time-symmetric systems. The proposed memory achieves topological noise immunity and all-optical state transitions simultaneously.

A Reconfigurable Convolution-in-Pixel CMOS Image Sensor Architecture

The separation of the data capture and analysis in modern vision systems has led to a massive amount of data transfer between the end devices and cloud computers, resulting in long latency, slow response, and high power consumption. Efficient hardware architectures are under focused development to enable Artificial Intelligence (AI) at the resource-limited sensing devices. One of the most promising solutions is to enable Processing-in-Pixel (PIP) scheme. However, the conventional schemes suffer from the low fill-factor issue. This paper proposes a PIP based Complementary Metal-Oxide-Semiconductor (CMOS) sensor architecture, which allows convolution operation before the column readout circuit to significantly reduce the overall power consumption while improving the resource utilization of the succeeding deep learning accelerator. The simulation results show that the proposed architecture could support the computing efficiency up to 3.37 TOPS/W at the 8-bit weight configuration, which is four times as high as the conventional schemes after normalization. The transistors required for each pixel are only 3.5T, significantly improving the fill-factor.

A Fine-Grained Modeling Approach for Systolic Array-Based Accelerator

The systolic array provides extremely high efficiency for running matrix multiplication and is one of the mainstream architectures of today’s deep learning accelerators. In order to develop efficient accelerators, people usually employ simulators to make design trade-offs. However, current simulators suffer from coarse-grained modeling methods and ideal assumptions, which limits their ability to describe structural characteristics of systolic arrays. In addition, they do not support the exploration of microarchitecture. This paper presents FG-SIM, a fine-grained modeling approach for evaluating systolic array accelerators by using an event-driven method. FG-SIM can obtain accurate results and provide the best mapping scheme for different workloads due to its fine-grained modeling technique and deny of ideal assumption. Experimental results show that FG-SIM plays a significant role in design trade-offs and outperforms state-of-the-art simulators, with an accuracy of more than 95%.

LiteCON : An All-photonic Neuromorphic Accelerator for Energy-efficient Deep Learning

Deep learning is highly pervasive in today's data-intensive era. In particular, convolutional neural networks (CNNs) are being widely adopted in a variety of fields for superior accuracy. However, computing deep CNNs on traditional CPUs and GPUs brings several performance and energy pitfalls. Several novel approaches based on ASIC, FPGA, and resistive-memory devices have been recently demonstrated with promising results. Most of them target only the inference (testing) phase of deep learning. There have been very limited attempts to design a full-fledged deep learning accelerator capable of both training and inference. It is due to the highly compute- and memory-intensive nature of the training phase. In this article, we propose LiteCON , a novel analog photonics CNN accelerator. LiteCON uses silicon microdisk-based convolution, memristor-based memory, and dense-wavelength-division-multiplexing for energy-efficient and ultrafast deep learning. We evaluate LiteCON using a commercial CAD framework (IPKISS) on deep learning benchmark models including LeNet and VGG-Net. Compared to the state of the art, LiteCON improves the CNN throughput, energy efficiency, and computational efficiency by up to 32×, 37×, and 5×, respectively, with trivial accuracy degradation.

An Efficient Deep Learning Accelerator Architecture for Compressed Video Analysis

Previous neural network accelerators tailored to video analysis only accept data of RGB/YUV domain, requiring decompressing the video that are often compressed before transmitted from the edge sensors. A compressed video processing accelerator can alleviate the decoding overhead, and gain performance speedup by operating on more compact input data. This work proposes a novel deep learning accelerator architecture, Alchemist, which is able to predict results directly from the compressed video bitstream instead of reconstructing the full RGB images. By utilizing the metadata of motion vector and critical blocks extracted from bitstreams, Alchemist contributes to a remarkable performance speedup of 5x with negligible accuracy loss. Nevertheless, we still find that the original compressed video coded by standard algorithms such as H.264 is not suitable to be directly manipulated, due to diverse compressed structures. Although obviating the requirement to recover all RGB frames, the accelerator must parse the entire compressed video bitstream to locate reference frames and extract useful metadata. If we combine the video codec with the proposed compressed video analysis, additional optimizations can be obtained. Therefore, to cope with the mismatch between current video coding algorithms, such as H.264 and neural network-based video analysis, we propose a specialized coding strategy to generate compressed video bitstreams more suitable for transmission and analysis, which further simplifies the decoding stage of video analysis and is capable of achieving significant storage reduction.

Int-Monitor: a model triggered hardware trojan in deep learning accelerators

Int-Monitor: a model triggered hardware trojan in deep learning accelerators

Higher order neural processing with input-adaptive dynamic weights on MoS2 memtransistor crossbars

The increasing complexity of deep learning systems has pushed conventional computing technologies to their limits. While the memristor is one of the prevailing technologies for deep learning acceleration, it is only suited for classical learning layers where only two operands, namely weights and inputs, are processed simultaneously. Meanwhile, to improve the computational efficiency of deep learning for emerging applications, a variety of non-traditional layers requiring concurrent processing of many operands are becoming popular. For example, hypernetworks improve their predictive robustness by simultaneously processing weights and inputs against the application context. Two-electrode memristor grids cannot directly map emerging layers’ higher-order multiplicative neural interactions. Addressing this unmet need, we present crossbar processing using dual-gated memtransistors based on two-dimensional semiconductor MoS2. Unlike the memristor, the resistance states of memtransistors can be persistently programmed and can be actively controlled by multiple gate electrodes. Thus, the discussed memtransistor crossbar enables several advanced inference architectures beyond a conventional passive crossbar. For example, we show that sneak paths can be effectively suppressed in memtransistor crossbars, whereas they limit size scalability in a passive memristor crossbar. Similarly, exploiting gate terminals to suppress crossbar weights dynamically reduces biasing power by ∼20% in memtransistor crossbars for a fully connected layer of AlexNet. On emerging layers such as hypernetworks, collocating multiple operations within the same crossbar cells reduces operating power by ∼15× on the considered network cases.