NNBench-X: A Benchmarking Methodology for Neural Network Accelerator Designs

Abstract

The tremendous impact of deep learning algorithms over a wide range of application domains has encouraged a surge of neural network (NN) accelerator research. Prior NN accelerator research reveals that the software-hardware co-design method is promising to achieve significant performance improvement and energy savings. To guide future co-designs, an evolving benchmark suite and its associated benchmark method are needed to characterize NN workloads and incorporate emerging NN compression techniques. However, the co-design method has not been well covered by existing NN benchmark work. In this paper, we propose a novel benchmarking methodology, which decouples the benchmarking process into three stages. First, we select the most representative applications from a user-customized candidate pool by analyzing their performance features. Second, we compress selected applications according to customized model compression techniques to generate a benchmark suite. Finally, we evaluate a variety of accelerator designs on the generated benchmark suite. To demonstrate the effectiveness of our benchmarking methodology, we conduct a case study of designing a general NN benchmark from TensorFlow Model Zoo and with various NN compression methods. We finally evaluate the benchmark on various representative architectures.