Mapping Neural Networks to FPGA-Based IoT Devices for Ultra-Low Latency Processing.

Maciej Wielgosz,Michał Karwatowski

doi:10.3390/s19132981

Abstract

Internet of things (IoT) infrastructure, fast access to knowledge becomes critical. In some application domains, such as robotics, autonomous driving, predictive maintenance, and anomaly detection, the response time of the system is more critical to ensure Quality of Service than the quality of the answer. In this paper, we propose a methodology, a set of predefined steps to be taken in order to map the models to hardware, especially field programmable gate arrays (FPGAs), with the main focus on latency reduction. Multi-objective covariance matrix adaptation evolution strategy (MO-CMA-ES) was employed along with custom scores for sparsity, bit-width of the representation and quality of the model. Furthermore, we created a framework which enables mapping of neural models to FPGAs. The proposed solution is validated using three case studies and Xilinx Zynq UltraScale+ MPSoC 285 XCZU15EG as a platform. The results show a compression ratio for quantization and pruning in different scenarios with and without retraining procedures. Using our publicly available framework, we achieved 210 ns of latency for a single processing step for a model composed of two long short-term memory (LSTM) and a single dense layer.

Highlights

Artificial Intelligence algorithms are developing rapidly and in multiple areas
This paper aims to introduce a methodology for mapping neural models to hardware field programmable gate arrays (FPGAs)
The high-level sequence of operations is very similar whether the flow in FPGAs after the module was mapped to hardware, or the one performed as an emulation before mapping, done on central processing unit (CPU), is considered

Summary

Introduction

Artificial Intelligence algorithms are developing rapidly and in multiple areas. One of the crucial factors causing this significant leap is the exponential increase of the available data, mainly due to the adoption of the IoTinternet of things (IoT). To be able to make correct decisions, from those vast amounts of data, the knowledge needs to be extracted. This task is usually done using machine learning algorithms, nowadays very often implemented as neural networks. Fast and well-defined system response time increasingly becomes a more decisive factor in the Quality of Service. The latency of the system response depends on both computing and data-transfer time between edge devices, over the network, and to the data centers. The most notable latency decrease can, be achieved by relocating as much of the computations into the edge (IoT) devices as possible. The amount of data transferred and processed in the upper levels of hierarchy can be significantly reduced

Objectives

Methods

Results

Discussion

Conclusion