Teacher−student framework‐based knowledge transfer with incremental block‐wise retraining

Abstract

This Letter proposes a stage-wise training method that uses block-wise retraining to transfer the useful knowledge of a pre-trained deep residual network (ResNet) in a teacher-student framework (TSF). To achieve this, flow-based hidden information transfer and hierarchically supervised retraining of the information are alternatively implemented from bottom to top between teacher and student ResNets in the TSF. To evaluate the effectiveness of the proposed method, the authors used well-known image data sets Canadian Institute For Advanced Research (CIFAR)-10, CIFAR-100, and street view house number. The results showed that the flow-based bottom-up knowledge transfer combined with incremental block-wise retraining can provide the improved small student ResNet with higher accuracy than the deep teacher ResNet. This approach will help extend the use of deep neural network models to limited computing environments.