Evaluating computing performance of deep neural network models with different backbones on IoT-based edge and cloud platforms

Abstract

This paper focuses on evaluating and predicting the computing performance of different architectures of deep neural network models (DNNs) in cross-platform and cross-inference frameworks. We test nearly 30 typical DNN models for image recognition on Google Colab cloud computing platform and Intel neural compute stick 2 embedded edge computing platform and record the computational performance metrics i.e. the Top-N accuracy, model complexity, computational complexity, inference time, memory usage, and so on. We compare and analyze these performance parameters with the previous workstation equipped with NVIDIA Titan X Pascal and an embedded system based on NVIDIA Jetson TX1 board to evaluate the inference efficiency of different DNN models using different inference frameworks. The methods of ANOVA are adopted to quantify the differences between the models. A combination method of cluster analysis and regression analysis is proposed to find the similar inference time variation processes across models, which can be used to predict the inference results of unknown models. These presented results will contribute to better deployment and application of resource-constrained DNN models on the heterogeneous high-performance computing platform.