Convolutional Neural Networks Inference Research Articles

One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search

Convolutional neural networks (CNNs) are used in numerous real-world applications such as vision-based autonomous driving and video content analysis. To run CNN inference on various target devices, hardware-aware neural architecture search (NAS) is crucial. A key requirement of efficient hardware-aware NAS is the fast evaluation of inference latencies in order to rank different architectures. While building a latency predictor for each target device has been commonly used in state of the art, this is a very time-consuming process, lacking scalability in the presence of extremely diverse devices. In this work, we address the scalability challenge by exploiting latency monotonicity --- the architecture latency rankings on different devices are often correlated. When strong latency monotonicity exists, we can re-use architectures searched for one proxy device on new target devices, without losing optimality. In the absence of strong latency monotonicity, we propose an efficient proxy adaptation technique to significantly boost the latency monotonicity. Finally, we validate our approach and conduct experiments with devices of different platforms on multiple mainstream search spaces, including MobileNet-V2, MobileNet-V3, NAS-Bench-201, ProxylessNAS and FBNet. Our results highlight that, by using just one proxy device, we can find almost the same Pareto-optimal architectures as the existing per-device NAS, while avoiding the prohibitive cost of building a latency predictor for each device.

Sparse convolutional neural network acceleration with lossless input feature map compression for resource‐constrained systems

Many recent research efforts have exploited data sparsity for the acceleration of convolutional neural network (CNN) inferences. However, the effects of data transfer between main memory and the CNN accelerator have been largely overlooked. In this work, the authors propose a CNN acceleration technique that leverages hardware/software co-design and exploits the sparsity in input feature maps (IFMs). On the software side, the authors' technique employs a novel lossless compression scheme for IFMs, which are sent to the hardware accelerator via direct memory access. On the hardware side, the authors' technique uses a CNN inference accelerator that performs convolutional layer operations with their compressed data format. With several design optimization techniques, the authors have implemented their technique in a field-programmable gate array (FPGA) system-on-chip platform and evaluated their technique for six different convolutional layers in SqueezeNet. Results reveal that the authors' technique improves the performance by 1.1×–22.6× while reducing energy consumption by 47.7%–97.4% as compared to the CPU-based execution. Furthermore, results indicate that the IFM size and transfer latency are reduced by 34.0%–85.2% and 4.4%–75.7%, respectively, compared to the case without data compression. In addition, the authors' hardware accelerator shows better performance per hardware resource with less than or comparable power consumption to the state-of-the-art FPGA-based designs.

Towards Edge Computing Using Early-Exit Convolutional Neural Networks

In computer vision applications, mobile devices can transfer the inference of Convolutional Neural Networks (CNNs) to the cloud due to their computational restrictions. Nevertheless, besides introducing more network load concerning the cloud, this approach can make unfeasible applications that require low latency. A possible solution is to use CNNs with early exits at the network edge. These CNNs can pre-classify part of the samples in the intermediate layers based on a confidence criterion. Hence, the device sends to the cloud only samples that have not been satisfactorily classified. This work evaluates the performance of these CNNs at the computational edge, considering an object detection application. For this, we employ a MobiletNetV2 with early exits. The experiments show that the early classification can reduce the data load and the inference time without imposing losses to the application performance.

A Multi-Cache System for On-Chip Memory Optimization in FPGA-Based CNN Accelerators

In recent years, FPGAs have demonstrated remarkable performance and contained power consumption for the on-the-edge inference of Convolutional Neural Networks. One of the main challenges in implementing this class of algorithms on board an FPGA is resource management, especially with regard to memory. This work presents a multi-cache system that allows for noticeably shrinking the required on-chip memory with a negligible variation of timing performance and power consumption. The presented methods have been applied to the CloudScout CNN, which was developed to perform cloud detection directly on board the satellite, thus representing a relevant case study for on the edge applications. The system was validated and characterized on a Xilinx ZCU106 Evaluation Board. The result is a 64.48% memory saving if compared to an alternative hardware accelerator developed for the same algorithm, with comparable performance in terms of inference time and power consumption. The paper also presents a detailed analysis of the hardware accelerator power consumption, focusing on the impact of data transfer between the accelerator and the external memory. Further investigation shows that the proposed strategies allow the implementation of the accelerator on FPGAs with a smaller size, guaranteeing benefits in terms of power consumption and hardware costs. A broader evaluation about the applicability of the presented methods to other models demonstrates valuable results in terms of memory saving with respect to other works reported in the literature.

Real-time on-site inspection system for power transmission based on heterogeneous computing

As the need of power supply is tremendously increasing in modern society, the stableness and reliability of the power delivery system are the two essential factors that ensure the power supply safety. With the quick expansion of electricity infrastructures, the failures of power transmission system are becoming more frequent, leading to economic loss and high risk of maintenance work under hazardous conditions. The existing automatic power line inspection utilizes advanced convolutional neural network (CNN) to improve the inspection efficiency, emerging as one promising solution. But the needed computational complexity is high since CNN inference demands large amount of multiplication-and-accumulation operations. In this paper, we alleviate this problem by utilizing the heterogeneous computing techniques to design a real-time on-site inspection system. Firstly, the required computational complexity of CNN inference is reduced using FFT-based convolution algorithms, speeding up the inference. Then we utilize the region of interest (ROI) extrapolation to predict the object detection bounding boxes without CNN inference, thus saving computing power. Finally, a heterogeneous computing architecture is presented to accommodate the requirements of proposed algorithms. According to the experiment results, the proposed design significantly improves the frame rate of CNN-based inspection visual system applied to power line inspection. The processing frame rate is also drastically improved. Moreover, the precision loss is negligible which means our proposed schemes are applicable for real application scenarios.

Simulating quantized inference on convolutional neural networks

Mobile and embedded applications of convolutional neural networks (CNNs) use quantization to reduce model size and increase computational efficiency. However, working with quantized networks often implies using non-standard training and execution methods, as modern frameworks offer limited support to fixed-point operations. We propose a quantization approach simulating the effects of quantization in CNN inference without needing to be re-implemented using fixed-point arithmetic, reducing overhead and complexity in evaluating existing networks’ responses to quantization. The proposed method provides a fast way of performing post-training quantization with different bit widths in activations and weights. Our experimental results on ImageNet CNNs show a model size reduction of more than 50%, while maintaining classification accuracy without a need for retraining. We also measured the relationship between classification complexity and tolerance to quantization, finding an inverse correlation between quantization level and dataset complexity.

Investigating data representation for efficient and reliable Convolutional Neural Networks

Nowadays, Convolutional Neural Networks (CNNs) are widely used as prediction models in different fields, with intensive use in real-time safety-critical systems. Recent studies have demonstrated that hardware faults induced by an external perturbation or aging effects, may significantly impact the CNN inference, leading to prediction failures. Therefore, ensuring the reliability of CNN platforms is crucial, especially when deployed in critical applications. A lot of effort has been made to reduce the memory and energy footprint of CNNs, paving the way to the adoption of approximate computing techniques such as quantization, reduced precision, weight sharing, and pruning. Unfortunately, approximate computing reduces the intrinsic redundancy of CNNs making them more efficient but less resilient to hardware faults. The goal of this work is twofold. First, we assess the reliability of a CNN when reduced bit widths and two different data types (floating- and fixed-point) are used to represent the network parameters (i.e., synaptic weights). Second, we intend to investigate the best compromise between data type, bit-widths reduction, and reliability. The characterization is performed through a fault injection environment built on the darknet open-source framework and targets two CNNs: LeNet-5 and YOLO. Experimental results show that fixed-point data provide the best trade-off between memory footprint reduction and CNN resilience. In particular, for LeNet-5, we achieved a 4X memory footprint reduction at the cost of a slightly reduced reliability (0.45% of critical faults) without retraining the CNN.

Accelerating Inference of Convolutional Neural Networks Using In-memory Computing.

In-memory computing (IMC) is a non-von Neumann paradigm that has recently established itself as a promising approach for energy-efficient, high throughput hardware for deep learning applications. One prominent application of IMC is that of performing matrix-vector multiplication in time complexity by mapping the synaptic weights of a neural-network layer to the devices of an IMC core. However, because of the significantly different pattern of execution compared to previous computational paradigms, IMC requires a rethinking of the architectural design choices made when designing deep-learning hardware. In this work, we focus on application-specific, IMC hardware for inference of Convolution Neural Networks (CNNs), and provide methodologies for implementing the various architectural components of the IMC core. Specifically, we present methods for mapping synaptic weights and activations on the memory structures and give evidence of the various trade-offs therein, such as the one between on-chip memory requirements and execution latency. Lastly, we show how to employ these methods to implement a pipelined dataflow that offers throughput and latency beyond state-of-the-art for image classification tasks.

A low‐cost compensated approximate multiplier for Bfloat16 data processing on convolutional neural network inference

This paper presents a low-cost two-stage approximate multiplier for bfloat16 (brain floating-point) data processing. For cost-efficient approximate multiplication, the first stage implements Mitchell's algorithm that performs the approximate multiplication using only two adders. The second stage adopts the exact multiplication to compensate for the error from the first stage by multiplying error terms and adding its truncated result to the final output. In our design, the low-cost multiplications in both stages can reduce hardware costs significantly and provide low relative errors by compensating for the error from the first stage. We apply our approximate multiplier to the convolutional neural network (CNN) inferences, which shows small accuracy drops with well-known pre-trained models for the ImageNet database. Therefore, our design allows low-cost CNN inference systems with high test accuracy.

An efficient loop tiling framework for convolutional neural network inference accelerators

Convolutional neural networks (CNNs) have been widely applied in the field of computer vision due to their inherent advantages in image feature extraction. However, it is difficult to implement CNNs directly on embedded platforms owing to excessive calculations of CNNs. Field Programmable Gate Arrays have been popular in CNN accelerators because of their configurability and high energy efficiency. Given the highly parallel workloads of the CNN, a CNN accelerator with a 14 × 16 processing element array is designed in this study to accelerate the CNN inference. Besides, a loop tiling strategy for convolutional layers is proposed to efficiently transmit feature maps. Additionally, the roofline model is employed to explore the best tiling parameters for optimal performance. Finally, the accelerator written in Verilog-HDL language is implemented on the Xilinx Zynq-7045 evaluation platform. At an operating frequency of 200 MHz, the proposed accelerator can achieve a performance of 57.24 giga operations per second on You Only Look Once v2-tiny and 78.39 GOPS on Visual Geometry Group-16. The accelerator only consumes 224 DSPs, demonstrating a better performance compared with the previous works.

Area and energy efficient shift and accumulator unit for object detection in IoT applications

Convolutional Neural Networks (CNNs) exhibit significant performance enhancements in several machine learning tasks such as surveillance, intelligent transportation, smart grids and healthcare systems. With the proliferation of physical things being connected to internet and enabled with sensory capabilities to form an Internet of Thing (IoT) network, it is increasingly important to run CNN inference, a computationally intensive application, on the resource constrained IoT devices. Object detection is a fundamental computer vision problem that provides information for image understanding in several artificial intelligence (AI) applications in smart cities. Among various object detection algorithms, CNN has emerged as a new paradigm to improve the overall performance. The Multiply-accumulate (MAC) operations, which are used repeatedly in the convolution layers of CNN, hold extreme computational complexity. Hence, the overall computational workloads and their respective energy consumption of any CNN applications are on the rise. To overcome these escalating challenges, approximate computing mechanism has played a vital role in reducing power and area of computation intensive CNN applications. In this paper, we have designed an approximate MAC architecture, termed Shift and Accumulator Unit (SAC), for the error-resilient CNN based object detection algorithm targeting embedded platforms. The proposed computing unit deliberately trades accuracy to reduce design complexity and power consumption, thus suiting the resource constrained IoT devices. The pipeline architecture of the SAC unit saves approximately 1.8× clock cycles than the non-pipeline SAC architecture. The performance evaluation shows that the proposed computing unit has better energy efficiency and resource utilization than the accurate multiplier and state-of-the-art approximate multipliers without noticeable deterioration in overall performance.

Methods for Preventing Visual Attacks in Convolutional Neural Networks Based on Data Discard and Dimensionality Reduction

The article is devoted to the study of convolutional neural network inference in the task of image processing under the influence of visual attacks. Attacks of four different types were considered: simple, involving the addition of white Gaussian noise, impulse action on one pixel of an image, and attacks that change brightness values within a rectangular area. MNIST and Kaggle dogs vs. cats datasets were chosen. Recognition characteristics were obtained for the accuracy, depending on the number of images subjected to attacks and the types of attacks used in the training. The study was based on well-known convolutional neural network architectures used in pattern recognition tasks, such as VGG-16 and Inception_v3. The dependencies of the recognition accuracy on the parameters of visual attacks were obtained. Original methods were proposed to prevent visual attacks. Such methods are based on the selection of “incomprehensible” classes for the recognizer, and their subsequent correction based on neural network inference with reduced image sizes. As a result of applying these methods, gains in the accuracy metric by a factor of 1.3 were obtained after iteration by discarding incomprehensible images, and reducing the amount of uncertainty by 4–5% after iteration by applying the integration of the results of image analyses in reduced dimensions.

SWM: A High-Performance Sparse-Winograd Matrix Multiplication CNN Accelerator

Many convolutional neural network (CNN) accelerators are proposed to exploit the sparsity of the networks recently to enjoy the benefits of both computation and memory reduction. However, most accelerators cannot exploit the sparsity of both activations and weights. For those works that exploit both sparsity opportunities, they cannot achieve the stable load balance through a static scheduling (SS) strategy, which is vulnerable to the sparsity distribution. In this work, a balanced compressed sparse row format and a dynamic scheduling strategy are proposed to improve the load balance. A set-associate structure is also presented to tradeoff the load balance and hardware resource overhead. We propose SWM to accelerate the CNN inference, which supports both sparse convolution and sparse fully connected (FC) layers. SWM provides Winograd adaptability for large convolution kernels and supports both 16-bit and 8-bit quantized CNNs. Due to the activation sharing, 8-bit processing can achieve theoretically twice the performance of the 16-bit processing with the same sparsity. The architecture is evaluated with VGG16 and ResNet50, which achieves: at most 7.6 TOP/s for sparse-Winograd convolution and three TOP/s for sparse matrix multiplication with 16-bit quantization on Xilinx VCU1525 platform. SWM can process 310/725 images per second for VGG16/ResNet50 with 16-bit quantization. Compared with the state-of-the-art works, our design can achieve at least 1.53 × speedup and 1.8 × energy efficiency improvement.

Toward Multi-FPGA Acceleration of the Neural Networks

High-throughput and low-latency Convolutional Neural Network (CNN) inference is increasingly important for many cloud- and edge-computing applications. FPGA-based acceleration of CNN inference has demonstrated various benefits compared to other high-performance devices such as GPGPUs. Current FPGA CNN-acceleration solutions are based on a single FPGA design, which are limited by the available resources on an FPGA. In addition, they can only accelerate conventional 2D neural networks. To address these limitations, we present a generic multi-FPGA solution, written in OpenCL, which can accelerate more complex CNNs (e.g., C3D CNN) and achieve a near linear speedup with respect to the available single-FPGA solutions. The design is built upon the Intel Deep Learning Accelerator architecture, with three extensions. First, it includes updates for better area efficiency (up to 25%) and higher performance (up to 24%). Second, it supports 3D convolutions for more challenging applications such as video learning. Third, it supports multi-FPGA communication for higher inference throughput. The results show that utilizing multiple FPGAs can linearly increase the overall bandwidth while maintaining the same end-to-end latency. In addition, the design can outperform other FPGA 2D accelerators by up to 8.4 times and 3D accelerators by up to 1.7 times.

S-CNN-ESystem: An end-to-end embedded CNN inference system with low hardware cost and hardware-software time-balancing

Abstract In recent years, a variety of dedicated hardware accelerators have been proposed using field programmable gate arrays (FPGAs) to accelerate convolutional neural networks (CNNs) on embedded platforms. However, the speed of dedicated hardware has always exceeded that of data acquisition/loading of software. Therefore, the hardware can stay idle due to a long wait for the software, which will lead to less benefits even when utilizing hardware acceleration. In this paper, a hardware-reusing strategy is proposed based on the structural characteristics of CNNs. Furthermore, a heuristic hardware/software time-balancing flow is introduced with the help of the hardware-reusing to narrow and even eliminate the imbalance of hardware-software time in the low-cost end-to-end embedded CNN inference system. On the premise that the system’s time and accuracy requirements are met, the hardware speed is optimized in hardware design to achieve the hardware/software time-balancing. Based on these two strategies, an end-to-end embedded CNN inference system with low hardware cost and hardware/software time-balancing (S-CNN-ESystem) is suggested. Evaluations have been performed on the Zynq xc7z020-clg400-1 platform and LeNet-5, AlexNet and VGG-16 used to validate our solution. Compared with the accelerators implemented by the pure FPGA, the hardware-to-software time consuming ratio is much closer to 1:1.

An FPGA-Based Hardware Accelerator for CNNs Inference on Board Satellites: Benchmarking with Myriad 2-Based Solution for the CloudScout Case Study

In recent years, research in the space community has shown a growing interest in Artificial Intelligence (AI), mostly driven by systems miniaturization and commercial competition. In particular, the application of Deep Learning (DL) techniques on board Earth Observation (EO) satellites might lead to numerous advantages in terms of mitigation of downlink bandwidth constraints, costs, and increment of the satellite autonomy. In this framework, the CloudScout project, funded by the European Space Agency (ESA), represents the first time in-orbit demonstration of a Convolutional Neural Network (CNN) applied to hyperspectral images for cloud detection. The first instance of this use case has been done with an INTEL Myriad 2 VPU on board a CubeSat optimized for low cost, size, and power efficiency. Nevertheless, this solution introduces multiple drawbacks due to its design not specifically being for the space environment, thus limiting its applicability to short-lifetime Low Earth Orbit (LEO) applications. The current work provides a benchmark between the Myriad 2 and our custom hardware accelerator designed for Field Programmable Gate Arrays (FPGAs). The metrics used for comparison include inference time, power consumption, space qualification, and components. The obtained results show that the FPGA-based solution is characterized by a reduced inference time, and a higher possibility of customization, but at the cost of greater power consumption and a longer Time to Market. As a conclusion, the proposed approach might extend the potential market of DL-based solutions to long-term LEO or interplanetary exploration missions through deployment on space-qualified FPGAs, with a limited cost in energy efficiency.

A Heterogeneous Hardware Accelerator for Image Classification in Embedded Systems

Convolutional neural networks (CNN) have been extensively employed for image classification due to their high accuracy. However, inference is a computationally-intensive process that often requires hardware acceleration to operate in real time. For mobile devices, the power consumption of graphics processors (GPUs) is frequently prohibitive, and field-programmable gate arrays (FPGA) become a solution to perform inference at high speed. Although previous works have implemented CNN inference on FPGAs, their high utilization of on-chip memory and arithmetic resources complicate their application on resource-constrained edge devices. In this paper, we present a scalable, low power, low resource-utilization accelerator architecture for inference on the MobileNet V2 CNN. The architecture uses a heterogeneous system with an embedded processor as the main controller, external memory to store network data, and dedicated hardware implemented on reconfigurable logic with a scalable number of processing elements (PE). Implemented on a XCZU7EV FPGA running at 200 MHz and using four PEs, the accelerator infers with 87% top-5 accuracy and processes an image of pixels in 220 ms. It consumes 7.35 W of power and uses less than 30% of the logic and arithmetic resources used by other MobileNet FPGA accelerators.

DeeperThings: Fully Distributed CNN Inference on Resource-Constrained Edge Devices

Performing inference of Convolutional Neural Networks (CNNs) on Internet of Things (IoT) edge devices ensures both privacy of input data and possible run time reductions when compared to a cloud solution. As most edge devices are memory- and compute-constrained, they cannot store and execute complex CNNs. Partitioning and distributing layer information across multiple edge devices to reduce the amount of computation and data on each device presents a solution to this problem. In this article, we propose DeeperThings, an approach that supports a full distribution of CNN inference tasks by partitioning fully-connected as well as both feature- and weight-intensive convolutional layers. Additionally, we jointly optimize memory, computation and communication demands. This is achieved using techniques to combine both feature and weight partitioning with a communication-aware layer fusion method, enabling holistic optimization across layers. For a given number of edge devices, the schemes are applied jointly using Integer Linear Programming (ILP) formulations to minimize data exchanged between devices, to optimize run times and to find the entire model’s minimal memory footprint. Experimental results from a real-world hardware setup running four different CNN models confirm that the scheme is able to evenly balance the memory footprint between devices. For six devices on 100 Mbit/s connections the integration of layer fusion additionally leads to a reduction of communication demands by up to 28.8%. This results in run time speed-up of the inference task by up to 1.52x compared to layer partitioning without fusing.

Mixed-Clipping Quantization for Convolutional Neural Networks

<p indent=0mm>Quantization is the main method to compress convolutional neural networks and accelerate convolutional neural network inference. Most existing quantization methods quantize all layers to the same bit width. Mixed-precision quantization can obtain higher precision under the same compression ratio, but it is difficult to find a mixed-precision quantization strategy. To solve this problem, a mixed-clipping quantization method based on reinforcement learning is proposed. It uses reinforcement learning to search for a mixed-precision quantization strategy, and uses a mixed-clipping method to clip weight data according to the searched quantization strategy before quantization. This method further improves the accuracy of the quantized network. We extensively test this method on a diverse set of models, including ResNet18/50, MobileNet-V2 on ImageNet, as well as YOLOV3 on the Microsoft COCO dataset. The experimental results show that our method can achieve 2.7% and 0.3% higher Top-1 accuracy on MobileNet-V2 (4 bit), as compared to the HAQ and ZeroQ method. And our method can achieve 2.6% higher mAP on YOLOV3 (6 bit), as compared to per-layer quantization method.

DeepSlicing: Collaborative and Adaptive CNN Inference With Low Latency

The booming of Convolutional Neural Networks (CNNs) has empowered lots of computer-vision applications. Due to its stringent requirement for computing resources, substantial research has been conducted on how to optimize its deployment and execution on resource-constrained devices. However, previous works have several weaknesses, including limited support for various CNN structures, fixed scheduling strategies, overlapped computations, high synchronization overheads, etc. In this article, we present DeepSlicing, a collaborative and adaptive inference system that adapts to various CNNs and supports customized flexible fine-grained scheduling. As a built-in functionality, DeepSlicing has supported typical CNNs including GoogLeNet, ResNet, etc. By partitioning both model and data, we also design an efficient scheduler, Proportional Synchronized Scheduler (PSS), which achieves the trade-off between computation and synchronization. Based on PyTorch, we have implemented DeepSlicing on the testbed with real-world edge settings that consists of 8 heterogeneous Raspberry Pi's. The results indicate that DeepSlicing with PSS outperforms the existing systems dramatically, e.g., the inference latency and memory footprint are reduced up to 5.79× and 14.72×, respectively.