Efficient image classification through collaborative knowledge distillation: A novel AlexNet modification approach

Abstract

This paper introduces an innovative image classification technique utilizing knowledge distillation, tailored for a lightweight model structure. The core of the approach is a modified version of the AlexNet architecture, enhanced with depthwise-separable convolution layers. A unique aspect of this work is the Teacher-Student Collaborative Knowledge Distillation (TSKD) method. Unlike conventional knowledge distillation techniques, TSKD employs a dual-layered learning strategy, where the student model learns from both the final output and the intermediate layers of the teacher model. This collaborative learning approach enables the student model to actively engage in the learning process, resulting in more efficient knowledge transfer. The paper emphasizes the model suitability for scenarios with limited computational resources. This is achieved through architectural optimizations and the introduction of specialized loss functions, which balance the trade-off between model complexity and computational efficiency. The study demonstrates that despite its lightweight nature, the model maintains high accuracy and robustness in image classification tasks. Key contributions of the paper include the innovative use of depthwise-separable convolution in AlexNet, the TSKD approach for enhanced knowledge transfer, and the development of unique loss functions. These advancements collectively contribute to the model effectiveness in environments with computational constraints, making it a valuable contribution to the field of image classification.