Model Compression for IoT Applications in Industry 4.0 via Multiscale Knowledge Transfer

Abstract

Recently, Industry 4.0 has attracted much attention. It has close relations with the Internet of Things (IoT). On the other hand, convolutional neural networks (CNNs) have shown promising performance in many foundational services of the IoT applications. For the IoT applications with high-speed data streams and the requirement of time-sensitive actions, fast processing is demanded on small-scale platforms or even on IoT devices themselves. Therefore, it is inappropriate to employ cumbersome CNNs in IoT applications, making the study of model compression necessary. In knowledge transfer, it is common to employ a deep, well-trained network, called teacher , to guide a shallow, untrained network, called student , to have better performance. Previous works have made many attempts to transfer single-scale knowledge from teacher to student , leading to degradation of generalization ability. In this article, we introduce multiscale representations to knowledge transfer, which facilitates the generalization ability of student . We divide student and teacher into several stages. Student learns from multiscale knowledge provided by teacher at the end of each stage. Extensive experiments demonstrate the effectiveness of our proposed method both on image classification and on single image super-resolution. The huge performance gap between student and teacher is significantly narrowed down by our proposed method, making student suitable for IoT applications.