Dynamic Neural Network Accelerator for Multispectral detection Based on FPGA

Abstract

Multispectral detection is an important and challenging task in computer vision. Two-stream architecture is commonly used in multispectral detection, which requires much more computational resources, more on-chip memory, and higher off-chip bandwidth than traditional CNNs. Compared to static neural networks, dynamic networks have advantages in terms of computational efficiency, adaptiveness, etc. In this paper, we propose an efficient dynamic neural network for multispectral detection. Compared with the unoptimized static model, our proposed model reduces computation by 41% and off-chip memory access by 77% on average with less than 1% decrease in mAP. We design cross-layer scheduling for dynamic networks that can fine-tune the fusion model at runtime. We implement a flexible FPGA-based hardware accelerator to evaluate our design. Experiments demonstrate that our dynamic neural network accelerator achieves a maximum 2.7× inference speedup.