Off-chip Memory Access Research Articles

FPGA-based tiling scheme and on-chip memory scheduling scheme for multi-branch semantic segmentation neural network accelerator

Abstract According to the characteristics of a multi-branch semantic segmentation neural network with a large amount of intermediate data during the inference process, we reduce the requirement of off-chip memory access and the redundant calculation due to tiling through better blocking scheme, better on-chip memory partitioning scheme, and on-chip memory scheduling scheme, improving the energy efficiency to 77.4 GOPS/W.

Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference

Recent advancements in large language models (LLMs) boasting billions of parameters have generated a significant demand for efficient deployment in inference workloads. While hardware accelerators for Transformer-based models have been extensively studied, the majority of existing approaches rely on temporal architectures that reuse hardware units for different network layers and operators. However, these methods often encounter challenges in achieving low latency due to considerable memory access overhead. This paper investigates the feasibility and potential of model-specific spatial acceleration for LLM inference on FPGAs. Our approach involves the specialization of distinct hardware units for specific operators or layers, facilitating direct communication between them through a dataflow architecture while minimizing off-chip memory accesses. We introduce a comprehensive analytical model for estimating the performance of a spatial LLM accelerator, taking into account the on-chip compute and memory resources available on an FPGA. This model can be extended to multi-FPGA settings for distributed inference. Through our analysis, we can identify the most effective parallelization and buffering schemes for the accelerator and, crucially, determine the scenarios in which FPGA-based spatial acceleration can outperform its GPU-based counterpart. To enable more productive implementations of an LLM model on FPGAs, we further provide a library of high-level synthesis (HLS) kernels that are composable and reusable. This library will be made available as open-source. To validate the effectiveness of both our analytical model and HLS library, we have implemented BERT and GPT2 on an AMD Xilinx Alveo U280 FPGA device. Experimental results demonstrate our approach can achieve up to 13.4 × speedup when compared to previous FPGA-based accelerators for the BERT model. For GPT generative inference, we attain a 2.2 × speedup compared to DFX, an FPGA overlay, in the prefill stage, while achieving a 1.9 × speedup and a 5.7 × improvement in energy efficiency compared to the NVIDIA A100 GPU in the decode stage.

Design of a Convolutional Neural Network Accelerator Based on On-Chip Data Reordering

Convolutional neural networks have been widely applied in the field of computer vision. In convolutional neural networks, convolution operations account for more than 90% of the total computational workload. The current mainstream approach to achieving high energy-efficient convolution operations is through dedicated hardware accelerators. Convolution operations involve a significant amount of weights and input feature data. Due to limited on-chip cache space in accelerators, there is a significant amount of off-chip DRAM memory access involved in the computation process. The latency of DRAM access is 20 times higher than that of SRAM, and the energy consumption of DRAM access is 100 times higher than that of multiply–accumulate (MAC) units. It is evident that the “memory wall” and “power wall” issues in neural network computation remain challenging. This paper presents the design of a hardware accelerator for convolutional neural networks. It employs a dataflow optimization strategy based on on-chip data reordering. This strategy improves on-chip data utilization and reduces the frequency of data exchanges between on-chip cache and off-chip DRAM. The experimental results indicate that compared to the accelerator without this strategy, it can reduce data exchange frequency by up to 82.9%.

Novel adaptive quantization methodology for 8-bit floating-point DNN training

There is a high energy cost associated with training Deep Neural Networks (DNNs). Off-chip memory access contributes a major portion to the overall energy consumption. Reduction in the number of off-chip memory transactions can be achieved by quantizing the data words to low data bit-width (E.g., 8-bit). However, low-bit-width data formats suffer from a limited dynamic range, resulting in reduced accuracy. In this paper, a novel 8-bit Floating Point (FP8) data format quantized DNN training methodology is presented, which adapts to the required dynamic range on-the-fly. Our methodology relies on varying the bias values of FP8 format to fit the dynamic range to the required range of DNN parameters and input feature maps. The range fitting during the training is adaptively performed by an online statistical analysis hardware unit without stalling the computation units or its data accesses. Our approach is compatible with any DNN compute cores without any major modifications to the architecture. We propose to integrate the new FP8 quantization unit in the memory controller. The FP32 data from the compute core are converted to FP8 in the memory controller before writing to the DRAM and converted back after reading the data from DRAM. Our results show that the DRAM access energy is reduced by 3.07×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ imes $$\\end{document} while using an 8-bit data format instead of using 32-bit. The accuracy loss of the proposed methodology with 8-bit quantized training is ≈1%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\approx 1\\%$$\\end{document} for various networks with image and natural language processing datasets.

Piezo-Acoustic Resistive Switching Behaviors in High-Performance Organic-Inorganic Hybrid Perovskite Memristors.

Memristors are regarded as promising candidates for breaking the problems including high off-chip memory access delays and the hash rate cost of frequent data moving induced by algorithms for data-intensive applications of existing computational systems. Recently, organic-inorganic halide perovskites (OIHPs) have been recognized as exceptionally favorable materials for memristors due to ease of preparation, excellent electrical conductivity, and structural flexibility. However, research on OIHP-based memristors focuses on modulating resistive switching (RS) performance through electric fields, resulting in difficulties in moving away from complex external circuits and wire connections. Here, a multilayer memristor has been constructed with eutectic gallium and indium (EGaIn)/ MAPbI3 /poly(3,4-ethylenedioxythiophene): poly(4-styrenesulphonate) (PEDOT: PSS)/indium tin oxide (ITO) structure, which exhibits reproducible and reliable bipolar RS with low SET/RESET voltages, stable endurance, ultrahigh average ON/OFF ratio, and excellent retention. Importantly, based on ion migration activated by sound-driven piezoelectric effects, the device exhibits a stable acoustic response with an average ON/OFF ratio greater than 103 , thus realizing non-contact, multi-signal, and far-field control in RS modulation. This study provides a single-structure multifunctional memristor as an integrated architecture for sensing, data storage, and computing.

CHIP-KNNv2: AConfigurable andHigh-PerformanceK-NearestNeighbors Accelerator on HBM-based FPGAs

The k-nearest neighbors (KNN) algorithm is an essential algorithm in many applications, such as similarity search, image classification, and database query. With the rapid growth in the dataset size and the feature dimension of each data point, processing KNN becomes more compute and memory hungry. Most prior studies focus on accelerating the computation of KNN using the abundant parallel resource on FPGAs. However, they often overlook the memory access optimizations on FPGA platforms and only achieve a marginal speedup over a multi-thread CPU implementation for large datasets.In this article, we design and implement CHIP-KNN: an HLS-based, configurable, and high-performance KNN accelerator. CHIP-KNN optimizes the off-chip memory access on modern HBM-based FPGAs such as the AMD/Xilinx Alveo U280 FPGA board. CHIP-KNN is configurable for all essential parameters used in the algorithm, including the size of the search dataset, the feature dimension and data type representation of each data point, the distance metric, and the number of nearest neighbors - K. In terms of design architecture, we explore and discuss the tradeoffs between two design versions: CHIP-KNNv1 (Ping-Pong buffer based) and CHIP-KNNv2 (streaming-based). Moreover, we investigate the routing congestion issue in our accelerator design, implement hierarchical structures to shorten critical paths, and integrate an open-source floorplanning optimization tool called TAPA/AutoBridge to eliminate the place-and-route issues. To explore the design space and balance the computation and memory access performance, we also build an analytical performance model. Given a user configuration of the KNN parameters, our tool can automatically generate TAPA HLS C code for the optimal accelerator design and the corresponding host code, on the HBM-based FPGA platform.Our experimental results on the Alveo U280 show that, compared to a 48-thread CPU implementation, CHIP-KNNv2 achieves a geomean performance speedup of 15×, with a maximum speedup of 45×. Additionally, we show that CHIP-KNNv2 achieves up to 2.1× performance speedup over CHIP-KNNv1 while increasing configurability. Compared with the state-of-the-art Facebook AI Similarity Search (FAISS) [23] GPU implementation running on a Nvidia Tesla V100 GPU, CHIP-KNNv2 achieves an average latency reduction of 30.6× while requiring 34.3% of GPU power consumption.

An Efficient Gustavson-Based Sparse Matrix–Matrix Multiplication Accelerator on Embedded FPGAs

Sparse-matrix sparse-matrix multiplication (SpMM) is an important kernel in multiple areas, e.g., data analytics and machine learning. Due to the low on-chip memory requirement, the consistent data format, and the simplified control logic, the Gustavson’s algorithm is a promising backbone algorithm for SpMM on hardware accelerators. However, the off-chip memory traffic still limits the performance of the algorithm, especially on embedded FPGAs. Previous researchers optimize the Gustavson’s algorithm targeting high bandwidth memory-based architectures and their solutions cannot be directly applied to embedded FPGAs with traditional DDRs. In this work, we propose an efficient Gustavson-based sparse matrix-matrix multiplication accelerator on embedded FPGAs. The proposed design fully considers the feature of off-chip memory access on embedded FPGAs and the dataflow of the Gustavson’s algorithm. At first, we analyze the parallelism of the algorithm and propose to perform the algorithm with element-wise parallelism, which reduces the idle time of processing elements caused by synchronization. Further, we show a counter-intuitive example that the traditional cache leads to worse performance. Then, we propose a novel access pattern-aware cache scheme called SpCache, which provides quick responses to reduce bank conflicts caused by irregular memory accesses and combines streaming and caching to handle requests that access ordered elements of unpredictable length. Moreover, we propose to perform the merge on part of partial results, which removes some redundant merges in the naive implementation and has little postprocessing overhead. Finally, we conduct experiments on the Xilinx Zynq-UltraScale ZCU106 platform with a set of benchmarks from the SuiteSparse matrix collection. The experimental results show that the proposed design achieves an average 1.75x performance speedup compared to the baseline.

Cuckoo Bloom Hybrid Filter: Algorithm and Hardware Architecture for High Performance Satellite Internet Protocol Route Lookup

The next-generation satellite Internet Protocol (IP) router is required to achieve tens of millions of route lookups per second, since satellite Internet services based on low Earth orbit (LEO) constellations have become a reality. Due to the limitation of hardware resources on satellites and the high reliability requirements for equipment, a new satellite IP route lookup architecture is proposed in this paper. The proposed architecture uses a Bloom and cuckoo filter-based structure called cuckoo Bloom hybrid filter (CBHF), which guarantees only one off-chip memory access per lookup, to accelerate the Prefix-Route Trie (PR-Trie) algorithm. The proposed architecture has been evaluated through both a behavioral simulation in C++ language and a hardware implementation in Verilog hardware description language (HDL). Our simulation and implementation results show that the proposed satellite IP route lookup architecture can achieve a single-port throughput beyond 13 Gbps on a field programmable gate array (FPGA) board with a single DDR3 memory chip when operating at 200 MHz. In addition, the resource utilization in the FPGA shows that the proposed architecture also supports triple modular redundancy (TMR) to enhance reliability.

An Ultra-Low Power TinyML System for Real-Time Visual Processing at Edge

Tiny machine learning (TinyML), executing AI workloads on resource and power strictly restricted systems, is an important and challenging topic. This brief firstly presents an extremely tiny backbone to construct high efficiency CNN models for various visual tasks. Then, a specially designed neural co-processor (NCP) is interconnected with MCU to build an ultra-low power TinyML system, which stores all features and weights on chip and completely removes both of latency and power consumption in off-chip memory access. Moreover, an application specific instruction-set is further presented for realizing agile development and rapid deployment. Extensive experiments demonstrate that the proposed TinyML system based on our tiny model, NCP and instruction set yields considerable accuracy and achieves a record ultra-low power of 160mW while implementing object detection and recognition at 30FPS. The demo video is available on https://www.youtube.com/watch?v=mIZPxtJ-9EY.

Implementation of Fully-Pipelined CNN Inference Accelerator on FPGA and HBM2 Platform

Many deep convolutional neural network (CNN) inference accelerators on the field-programmable gate array (FPGA) platform have been widely adopted due to their low power consumption and high performance. In this paper, we develop the following to improve performance and power efficiency. First, we use a high bandwidth memory (HBM) to expand the bandwidth of data transmission between the off-chip memory and the accelerator. Second, a fully-pipelined manner, which consists of pipelined inter-layer computation and a pipelined computation engine, is implemented to decrease idle time among layers. Third, a multi-core architecture with shared-dual buffers is designed to reduce off-chip memory access and maximize the throughput. We designed the proposed accelerator on the Xilinx Alveo U280 platform with in-depth Verilog HDL instead of high-level synthesis as the previous works and explored the VGG-16 model to verify the system during our experiment. With a similar accelerator architecture, the experimental results demonstrate that the memory bandwidth of HBM is 13.2× better than DDR4. Compared with other accelerators in terms of throughput, our accelerator is 1.9×/1.65×/11.9× better than FPGA+HBM2 based/low batch size (4) GPGPU/low batch size (4) CPU. Compared with the previous DDR+FPGA/DDR+GPGPU/DDR+CPU based accelerators in terms of power efficiency, our proposed system provides 1.4-1.7×/1.7-12.6×/6.6-37.1× improvement with the large-scale CNN model.

Eidetic: An In-Memory Matrix Multiplication Accelerator for Neural Networks

This paper presents the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Eidetic</i> architecture, which is an SRAM-based ASIC neural network accelerator that eliminates the need to continuously load weights from off-chip, while also minimizing the need to go off chip for intermediate results. Using in-situ arithmetic in the SRAM arrays, this architecture can supports a variety of precision types allowing for effective inference. We also present different data mapping policies for matrix-vector based networks (RNN and MLP) on the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Eidetic</i> architecture and describe the tradeoffs involved. With this architecture, multiple layers of a network can be concurrently mapped, storing both the layer weights and intermediate results on-chip, removing the energy and latency penalty of off-chip memory accesses. We evaluate <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Eidetic</i> on Google's Neural Machine Translation System (GNMT) encoder and demonstrate a 17.20× increase in throughput and 7.77× reduction in average latency over a single TPUv2 chip.

TranCIM: Full-Digital Bitline-Transpose CIM-based Sparse Transformer Accelerator With Pipeline/Parallel Reconfigurable Modes

Transformer models achieve excellent results in the fields like natural language processing, computer vision, and bioinformatics. Their large numbers of matrix multiplications (MMs) lead to substantial data movement and computation. Although computing-in-memory (CIM) has proven to be an efficient architecture for MM computation, transformer’s attention mechanism raises new challenges in memory access and computation aspects: the dynamic MM in attention layers causes redundant off-chip memory access; Attention layers dominate transformer’s computation and require high precision. Thus, we design a bitline-transpose CIM-based transformer accelerator TranCIM with pipeline/parallel reconfigurable modes. The pipeline mode alleviates off-chip access for attention layers. The parallel mode is used by fully-connected (FC) layers for high parallelism. The full-digital CIM supports INT16 for attention layers and INT8 for FC layers, without analog CIM’s nonideal issues. Moreover, a sparse attention scheduler (SAS) is proposed to reduce attention computation. The fabricated TranCIM chip only consumes 15.59 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mu$</tex-math> </inline-formula> J/Token for the bidirectional encoder representations from transformer (BERT)-base model, achieving 12.08 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times$</tex-math> </inline-formula> –36.82 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times$</tex-math> </inline-formula> lower energy than prior CIM-based accelerators.

Re-Cache: Mitigating cache contention by exploiting locality characteristics with reconfigurable memory hierarchy for GPGPUs

Modern GPGPUs have employed multi-threading to hide the long off-chip memory access latency caused by frequent cache misses. However, the limited cache capacity shared by thousands of concurrently running warps will introduce serious cache contention problem, which results in high cache miss rates to hurt the overall GPGPU performance. To address this problem, this paper proposes a novel cache technology (Re-Cache) by exploiting locality characteristics and reconfiguring the memory hierarchy including the unused shared memory space and the idle registers of pending warps without extra storage overhead for GPGPUs. To further improve the utilization efficiency of the Re-Cache space, this paper proposes a dynamic and reconfigurable cache organization for Re-Cache that explores the spatial and temporal locality differences and changes among the running applications. Re-Cache dynamically reconfigures the cache organization as the adaptive prefetch region and victim region according to the differences and changes of the spatial and temporal locality by catching the locality characteristics for each load instruction. For the high-locality load instructions, the prefetch region stores the subsequent data blocks to exploit the spatial locality of the running threads among all streaming multiprocessors and warps, while the victim region collects the evicted data blocks from the upper memory hierarchy to exploit the temporal locality. Experimental results demonstrate that the proposed Re-Cache reduces up to 34.82% and 31.62% cache misses, while improving about 47.17% and 14.96% IPC performance compared with the state-of-the-art designs for cache-sensitive kernels and benchmarks, respectively.

PR-Trie: A Hybrid Trie with Ant Colony Optimization Based Prefix Partitioning for Memory-Efficient IPv4/IPv6 Route Lookup

Memory-efficient Internet Protocol (IP) lookup with high speed is essential to achieve link-speed packet forwarding in IP routers. The rapid growth of Internet traffic and the development of optical link technologies have made IP lookup a major performance bottleneck in core routers. In this paper, we propose a new IP route lookup architecture based on hardware called Prefix-Route Trie (PR-Trie), which supports both IPv4 and IPv6 addresses. In PR-Trie, we develop a novel structure called Overlapping Hybrid Trie (OHT) to perform fast longest-prefix-matching (LPM) based on Multibit-Trie (MT), and a hash-based level matching query used to achieve only one off-chip memory access per lookup. In addition, the proposed PR-Trie also supports fast incremental updates. Since the memory complexity in MT-based IP lookup schemes depends on the level-partitioning solution and the data structure used, we develop an optimization algorithm called Bitmap-based Prefix Partitioning Optimization (BP2O). The proposed BP2O is based on a heuristic search using Ant Colony Optimization (ACO) algorithms to optimize memory efficiency. Experimental results using real-life routing tables prove that our proposal has superior memory efficiency. Theoretical performance analyses show that PR-Trie outperforms the classical Trie-based IP lookup algorithms.

A High-Performance FPGA-Based Depthwise Separable Convolution Accelerator

Depthwise separable convolution (DSC) significantly reduces parameter and floating operations with an acceptable loss of accuracy and has been widely used in various lightweight convolutional neural network (CNN) models. In practical applications, however, DSC accelerators based on graphics processing units (GPUs) cannot fully exploit the performance of DSC and are unsuitable for mobile application scenarios. Moreover, low resource utilization due to idle engines is a common problem in DSC accelerator design. In this paper, a high-performance DSC hardware accelerator based on field-programmable gate arrays (FPGAs) is proposed. A highly reusable and scalable multiplication and accumulation engine is proposed to improve the utilization of computational resources. An efficient convolution algorithm is proposed for depthwise convolution (DWC) and pointwise convolution (PWC), respectively, to reduce the on-chip memory occupancy. Meanwhile, the proposed convolution algorithms achieve partial fusion between PWC and DWC, and improve the off-chip memory access efficiency. To maximise bandwidth utilization and reduce latency when reading feature maps, an address mapping method for off-chip accesses is proposed. The performance of the proposed accelerator is demonstrated by implementing MobileNetV2 on an Intel Arria 10 GX660 FPGA by using Verilog HDL. The experimental results show that the proposed DSC accelerator achieves a performance of 205.1 FPS, 128.8 GFLOPS, and 0.24 GOPS/DSP for input images of size 224×224×3.

FPSNET: An Architecture for Neural-Network-Based Feature Point Extraction for SLAM

The hardware architecture of a deep-neural-network-based feature point extraction method is proposed for the simultaneous localization and mapping (SLAM) in robotic applications, which is named the Feature Point based SLAM Network (FPSNET). Some key techniques are deployed to improve the hardware and power efficiency. The data path is devised to reduce overall off-chip memory accesses. The intermediate data and partial sums resulting in the convolution process are stored in available on-chip memories, and optimized hardware is employed to compute the one-point activation function. Meanwhile, address generation units are used to avoid data overlapping in memories. The proposed FPSNET has been designed in 65 nm CMOS technology with a core area of 8.3 mm2. This work reduces the memory overhead by 50% compared to traditional data storage for activation and overall by 35% for on-chip memories. The synthesis and simulation results show that it achieved a 2.0× higher performance compared with the previous design while achieving a power efficiency of 1.0 TOPS/W, which is 2.4× better than previous work. Compared to other ASIC designs with close peak throughput and power efficiency performance, the presented FPSNET has the smallest chip area (at least 42.4% reduction).

Omega-Test: A Predictive Early-Z Culling to Improve the Graphics Pipeline Energy-Efficiency.

The most common task of GPUs is to render images in real time. When rendering a 3D scene, a key step is to determine which parts of every object are visible in the final image. There are different approaches to solve the visibility problem, the Z-Test being the most common. A main factor that significantly penalizes the energy efficiency of a GPU, especially in the mobile arena, is the so-called overdraw, which happens when a portion of an object is shaded and rendered but finally occluded by another object. This useless work results in a waste of energy; however, a conventional Z-Test only avoids a fraction of it. In this article we present a novel microarchitectural technique, the Omega-Test, to drastically reduce the overdraw on a Tile-Based Rendering (TBR) architecture. Graphics applications have a great degree of inter-frame coherence, which makes the output of a frame very similar to the previous one. The proposed approach leverages the frame-to-frame coherence by using the resulting information of the Z-Test for a tile (a buffer containing all the calculated pixel depths for a tile), which is discarded by nowadays GPUs, to predict the visibility of the same tile in the next frame. As a result, the Omega-Test early identifies occluded parts of the scene and avoids the rendering of non-visible surfaces eliminating costly computations and off-chip memory accesses. Our experimental evaluation shows average EDP savings in the overall GPU/Memory system of 26.4 percent and an average speedup of 16.3 percent for the evaluated benchmarks.

Medha: Microcoded Hardware Accelerator for computing on Encrypted Data

Homomorphic encryption enables computation on encrypted data, and hence it has a great potential in privacy-preserving outsourcing of computations to the cloud. Hardware acceleration of homomorphic encryption is crucial as software implementations are very slow. In this paper, we present design methodologies for building a programmable hardware accelerator for speeding up the cloud-side homomorphic evaluations on encrypted data.First, we propose a divide-and-conquer technique that enables homomorphic evaluations in the polynomial ring RQ,2N = ZQ[x]/(x2N + 1) to use a hardware accelerator that has been built for the smaller ring RQ,N = ZQ[x]/(xN + 1). The technique makes it possible to use a single hardware accelerator flexibly for supporting several homomorphic encryption parameter sets.Next, we present several architectural design methods that we use to realize the flexible and instruction-set accelerator architecture, which we call ‘Medha’. At every level of the implementation hierarchy, we explore possibilities for parallel processing. Starting from hardware-friendly parallel algorithms for the basic building blocks, we gradually build heavily parallel RNS polynomial arithmetic units. Next, many of these parallel units are interconnected elegantly so that their interconnections require the minimum number of nets, therefore making the overall architecture placement-friendly on the platform. As homomorphic encryption is computation- as well as data-centric, the speed of homomorphic evaluations depends greatly on the way the data variables are handled. For Medha, we take a memory-conservative design approach and get rid of any off-chip memory access during homomorphic evaluations.Finally, we implement Medha in a Xilinx Alveo U250 FPGA and measure timing performances of the microcoded homomorphic addition, multiplication, key-switching, and rescaling routines for the leveled fully homomorphic encryption scheme RNSHEAAN at 200 MHz clock frequency. For the large parameter sets (log Q,N) = (438, 214) and (546, 215), Medha achieves accelerations by up to 68× and 78× times respectively compared to a highly optimized software implementation Microsoft SEAL running at 2.3 GHz.

FitNN: A Low-Resource FPGA-Based CNN Accelerator for Drones

Executing deep neural networks (DNNs) on resource-constraint edge devices, such as drones, offers low inference latency, high data privacy, and reduced network traffic. However, deploying DNNs on such devices is a challenging task. During DNN inference, intermediate results require significant data movement and frequent off-chip memory (DRAM) access, which decreases the inference speed and power efficiency. To address this issue, this article presents a field-programmable gate array (FPGA)-based convolutional neural network (CNN) accelerator, named FitNN, which improves the speed and power efficiency of CNN inference by reducing data movements. FitNN adopts a pretrained CNN of iSmart2, which is composed of depthwise and pointwise blocks in the Mobilenet structure. A cross-layer dataflow strategy is proposed to reduce off-chip data transfer of feature maps. Also, multilevel buffers are proposed to keep the most needed data on-chip (in block RAM) and avoid off-chip data reorganization and reloading. Finally, a computation core is proposed to operate the depthwise, pointwise, and max-pooling computation as soon as the data arrive without reorganization, which suits the real-life scenario of the data arriving in sequence. In our experiment, FitNN is implemented on two FPGA-based platforms (both at 150 MHz), Ultra96-V2 and PYNQ-Z1, for drone-based object detection with batch size = 1. The results show that FitNN achieves 15 frames per second (FPS) on Ultra96-V2, with power consumption of 4.69 W. On PYNQ-Z1, FitNN achieves 9 FPS with 1.9 W of power consumption. Compared with the previous FPGA-based implementation of iSmart2 CNN, FitNN increases the efficiency (FPS/W) by 2.37 times.

Optimus: An Operator Fusion Framework for Deep Neural Networks

The reduction of neural parameters and operations for the applications on embedded and IoT platforms in current deep neural network (DNN) architectures has received increasing attention. Relatively, the intermediate feature maps of such lightweight neural networks begin to grow and usually outsize the on-chip memory as the new bottleneck, which introduces considerable power-consuming off-chip memory accesses. To reduce the feature-induced memory accesses, operator fusion has been proposed to parallelize the execution of multiple convolutional layers and shown significant reduction of off-chip memory accesses. However, how to fuse the neural operators is still a challenging issue that heavily depends on both the neural network (NN) topology and the specific DNN accelerator configuration. In this work, we observed prior operator fusion approaches fail to guarantee memory-level optimality as they search in the constrained operator fusion design space. Considering the complexity of the NN topologies and the constrained resources of the DNN accelerators, we develop a novel operator fusion framework, Optimus. Optimus includes an accurate memory cost model dedicated to the scheduler to evaluate the potential operator-fusion schemes and a directed acyclic graph-based operator fusion algorithm for both off-line and on-line workload deployment scenarios, which altogether generates high-efficiency operator-fusion solutions for arbitrary network models running on DNN accelerators. The experimental results show that Optimus reduces 17–75% off-chip memory accesses and obtains 1.86×–3.66× energy efficiency on state-of-the-art DNN workloads when compared to the baselines and brings significant power-efficiency boost to the DNN accelerators of different architectures and dataflows.