Knowledge distillation-based deep learning classification network for peripheral blood leukocytes

Abstract

Leukocytes are a critical component of the human immune system. Many diseases can be diagnosed by analyzing the morphology and number of leukocytes. Due to the extensive application of convolutional neural networks (CNNs) in computer vision (CV), computer-aided automated methods have become the preferred methods for medical image diagnoses. Recently, Transformer has emerged in CV with performance comparable to CNN. Assisted diagnoses are often performed on resource-limited computing devices. The deployments of deep learning (DL) models are limited by the number of parameters and the computation. This study provides a DL training framework that introduces a model compression method of knowledge distillation (KD) in the classification of leukocytes, using small models instead of large ones, to achieve accurate results. Firstly, large models with CNN or Transformer structure are pre-trained on the mixed leukocyte dataset with 25,830 original images. Then, the dark knowledge of the pre-trained large models is extracted by KD, and the small models are trained. Finally, the best performing small model is selected as the final prediction model, which achieves 98.31% testing accuracy on the mixed dataset. The proposed framework on the enhanced BCCD dataset achieves 99.88% testing accuracy, which is better than other methods. It effectively combines the advantages of large and small models to meet the requirements of low resource consumption and high accuracy.