Knowledge Distillation Method Research Articles

Multiloss Joint Gradient Control Knowledge Distillation for Image Classification

Knowledge distillation (KD) techniques aim to transfer knowledge from complex teacher neural networks to simpler student networks. In this study, we propose a novel knowledge distillation method called Multiloss Joint Gradient Control Knowledge Distillation (MJKD), which functions by effectively combining feature- and logit-based knowledge distillation methods with gradient control. The proposed knowledge distillation method discretely considers the gradients of the task loss (cross-entropy loss), feature distillation loss, and logit distillation loss. The experimental results suggest that logits may contain more information and should, consequently, be assigned greater weight during the gradient update process in this work. The empirical findings on the CIFAR-100 and Tiny-ImageNet datasets indicate that MJKD generally outperforms traditional knowledge distillation methods, significantly enhancing the generalization ability and classification accuracy of student networks. For instance, MJKD achieves a 63.53% accuracy on Tiny-ImageNet for the ResNet18 MobileNetV2 pair. Furthermore, we present visualizations and analyses to explore its potential working mechanisms.

Uncertainty-based knowledge distillation for Bayesian deep neural network compression

Deep learning models have been widely employed across various fields. In real-world scenarios, especially safety-critical applications, quantifying uncertainty is as crucial as achieving high accuracy. To address this concern, Bayesian deep neural networks (BDNNs) emerged to estimate two different types of uncertainty: Aleatoric and Epistemic. Nevertheless, implementing a BDNN on resource-constrained devices poses challenges due to the substantial computational and storage costs imposed by approximation inference techniques. Thus, efficient compression methods should be utilized. We propose an uncertainty-based knowledge distillation method to compress BDNNs. Knowledge distillation is a model compression technique that involves transferring knowledge from a complex network, known as the teacher network, to a simpler one, referred to as the student network. Our method incorporates uncertainty into knowledge distillation to address situations where inappropriate teacher supervision undermines compression performance. We utilize the Epistemic uncertainty of teacher predictions to tailor supervision for each sample individually to take into account teacher's limited knowledge. Additionally, we adjust the temperature parameter of the distillation process for each sample based on the Aleatoric uncertainty of the teacher predictions, ensuring that the student receives appropriate supervision even in the presence of ambiguous data. As a result, the proposed method enables the Bayesian student network to be trained under both appropriate supervision of the Bayesian teacher network and ground truth labels. We evaluated our method on the CIFAR-10, CIFAR-100, and RAF-DB datasets, demonstrating notable improvements in accuracy over state-of-the-art knowledge distillation-based methods. Furthermore, the robustness of our approach was assessed through testing weakly trained teacher networks and the analysis of blurred and low-resolution data, which have high uncertainty. Experimental results show that the proposed method outperformed existing methods.

A Knowledge Distillation Method Based on Evidence Theory to Prevent Catastrophic Forgetting

Abstract Incremental learning is an emerging machine-learning approach designed to prevent catastrophic forgetting while learning a new task with knowledge retained from previous tasks. However, existing methods often must fully utilize the distributional information of the old task model’s outputs. To alleviate this, we propose a class incremental learning method grounded in evidence theory to leverage the distributional information of outputs from the old task model, which integrates uncertainty estimation into the knowledge distillation process to ensure the new task model effectively learns from the old task model’s distribution information. Specifically, we incorporate uncertainty estimation based on evidence theory to calculate knowledge distillation loss during training. We compute the uncertainty estimates of outputs from both the new and old task models, and then they are used in the knowledge distillation loss calculation. We propose a novel classification strategy that considers outputs’ probability and uncertainty estimates, which could determine the sample category without requiring an additional training phase or supplementary models. Experiments on the CIFAR-100 and ImageNet datasets demonstrate the effectiveness of the proposed method, showing that it outperforms existing methods by improving accuracy by 1%-2%. Results suggest that leveraging the distribution information of model outputs can effectively mitigate catastrophic forgetting in deep learning models within incremental learning scenarios.

Retrieval In Decoder benefits generative models for explainable complex question answering

Large-scale Language Models (LLMs) utilizing the Chain-of-Thought prompting demonstrate exceptional performance in a variety of tasks. However, the persistence of factual hallucinations remains a significant challenge in practical applications. Prevailing retrieval-augmented methods treat the retriever and generator as separate components, which inadvertently restricts the generator’s capabilities to those of the retriever through intensive supervised training. In this work, we propose an unsupervised Retrieval In Decoder framework for multi-granularity decoding called RID, which integrates retrieval directly into the decoding process of generative models. It dynamically adjusts decoding granularity based on retrieval outcomes, and duly corrects the decoding direction through its direct impact on the next token. Moreover, we introduce a reinforcement learning-driven knowledge distillation method for adaptive explanation generation to better apply to Small-scale Language Models (SLMs). The experimental results across six public benchmarks surpass popular LLMs and existing retrieval-augmented methods, which demonstrates the effectiveness of RID in models of different scales and verifies its applicability and scalability.

AssocKD: An Association-Aware Knowledge Distillation Method for Document-Level Event Argument Extraction

Event argument extraction is a crucial subtask of event extraction, which aims at extracting arguments that correspond to argument roles when given event types. The majority of current document-level event argument extraction works focus on extracting information for only one event at a time without considering the association among events; this is known as document-level single-event extraction. However, the interrelationship among arguments can yield mutual gains in their extraction. Therefore, in this paper, we propose AssocKD, an Association-aware Knowledge Distillation Method for Document-level Event Argument Extraction, which enables the enhancement of document-level multi-event extraction with event association knowledge. Firstly, we introduce an association-aware training task to extract unknown arguments with the given privileged knowledge of relevant arguments, obtaining an association-aware model that can construct both intra-event and inter-event relationships. Secondly, we adopt multi-teacher knowledge distillation to transfer such event association knowledge from the association-aware teacher models to the event argument extraction student model. Our proposed method, AssocKD, is capable of explicitly modeling and efficiently leveraging event association to enhance the extraction of multi-event arguments at the document level. We conduct experiments on RAMS and WIKIEVENTS datasets and observe a significant improvement, thus demonstrating the effectiveness of our method.

Efficient face anti-spoofing via head-aware transformer based knowledge distillation with 5 MB model parameters

Although face recognition technology has been applied in many scenarios, it still suffers from many types of presentation attacks, so face anti-spoofing (FAS) becomes a hot topic in computer vision. Recently, vision transformer is recognized as the mainstream architecture for FAS, which always relies on auxiliary information, sophisticated tricks and huge model parameters. Considering that face based identity authentication usually takes place on mobile-like devices, therefore how to design an effective and lightweight model is of great significance. Inspired by the powerful global modeling ability of self-attention and the model compression ability of knowledge distillation, a simple yet effective knowledge distillation approach is proposed for FAS under transformer framework. Our primary idea is to leverage the rich knowledge of a teacher network pre-trained on large-scale face data to guide the learning of a lightweight student network. The main contributions of our method are threefold: (1) Feature- and logits-level distillation are combined to transfer the rich knowledge of teacher to student. (2) A head-aware strategy is proposed to deal with the dimension mismatching issue of middle encoder layers between teacher and student networks, in which a novel attention head correlation matrix is introduced. (3) Our method can bridge the performance gap between teacher and student, and the resulting student network is extremely lightweight with only 5 MB parameters. Extensive experiments are conducted on three public face-spoofing datasets, CASIA-FASD, Replay-Attack and OULU-NPU, the results demonstrate that our method can obtain performance on par with or superior to most FAS methods and outperform many knowledge distillation methods. Meanwhile, the distilled student network achieves excellent performance with 17× fewer parameters and 9× faster inference time compared to the teacher network. The code will be publicly available at https://github.com/Maricle-zhangjun/HaTFAS.

Efficient skin lesion segmentation with boundary distillation.

Medical image segmentation models are commonly known for their complex structures, which often render them impractical for use on edge computing devices and compromising efficiency in the segmentation process. In light of this, the industry has proposed the adoption of knowledge distillation techniques. Nevertheless, the vast majority of existing knowledge distillation methods are focused on the classification tasks of skin diseases. Specifically, for the segmentation tasks of dermoscopy lesion images, these knowledge distillation methods fail to fully recognize the importance of features in the boundary regions of lesions within medical images, lacking boundary awareness for skin lesions. This paper introduces pioneering medical image knowledge distillation architecture. The aim of this method is to facilitate the efficient transfer of knowledge from existing complex medical image segmentation networks to a more simplified student network. Initially, a masked boundary feature (MBF) distillation module is designed. By applying random masking to the periphery of skin lesions, the MBF distillation module obliges the student network to reproduce the comprehensive features of the teacher network. This process, in turn, augments the representational capabilities of the student network. Building on the MBF distillation module, this paper employs a cascaded combination approach to integrate the MBF distillation module into a multi-head boundary feature (M2BF) distillation module, further strengthening the student network's feature learning ability and enhancing the overall image segmentation performance of the distillation model. This method has been experimentally validated on the public datasets ISIC-2016 and PH2, with results showing significant performance improvements in the student network. Our findings highlight the practical utility of the lightweight network distilled using our approach, particularly in scenarios demanding high operational speed and minimal storage usage. This research offers promising prospects for practical applications in the realm of medical image segmentation.

Data Tells the Truth: A Knowledge Distillation Method for Genomic Survival Analysis by Handling Censoring

Survival analysis is a critical tool for cancer research, yet handling censored data remains challenging due to supervision bias and inaccurate hazard estimates. To address these issues, we propose a simple but effective method termed KD, which employs knowledge distillation using uncensored data to rectify the supervision bias in censored data. This approach leverages the combined power of both rectified censored data and uncensored data to improve survival prediction accuracy. Remarkably, our KD method not only effectively harnesses censored data but also better reflects clinical reality, demonstrating its immense value in survival analysis. We applied our KD method to 19 target cancer sites using The Cancer Genome Atlas (TCGA) dataset. Our results consistently outperform traditional machine learning and deep learning-based methods across both target cancer sites and independent cancer cohorts. More importantly, our data-driven approach enables the model to extract hidden information from censored data, leading to conclusions that align more closely with clinical knowledge and scenarios. This validation of our KD method’s effectiveness highlights the substantial value of rational censored data usage, providing valuable insights for cancer research and clinical decisions. All data and codes are freely available at: https://datatellstruth.github.io/.

A detection method for dead caged hens based on improved YOLOv7

In large-scale laying hen farms, daily inspection of dead hens is a relevant task to monitor the health of the flock and prevent disease spreading. The current manual inspection method used in caged hen farms is inefficient, costly, and particularly difficult for high-rise cages. To address this issue, a dead caged hen detection method based on improved You Only Look Once version 7 (YOLOv7) was proposed in this study, which was optimized to improve detection performance and speed in complex farming environments, such as cage wire mesh occlusion and crowded hen occlusion. First, the Convolutional Block Attention Module was used to enable the model to learn target features accurately. Second, the Distance Intersection over Union Non-maximum Suppression and repulsion loss were introduced to improve crowded hen occlusion and reduce missed detections. Additionally, to facilitate the deployment of the proposed method on mobile devices, the MobileNetv3 lightweight network was used to replace the backbone of YOLOv7. Furthermore, the lightweight model was trained using the knowledge distillation method to enhance its performance. Finally, a comparison experiment of different object detection networks and an ablation experiment were conducted to evaluate the proposed method. The experimental results reveal that the improved YOLOv7 model proposed in this study performs optimally. Its precision, recall, F1 score, and mAP@0.5 for the dead hens in the test set are 95.7 %, 86.8 %, 0.910, and 86.2 %, respectively. Compared with the original YOLOv7 model, precision, recall, and mAP@0.5 were increased by 6 %, 10 %, and 13.4 %, respectively. The model parameters and Giga Floating-point Operations were decreased by 31.95 % and 60.56 %, respectively, resulting in a detection speed increase of 43 Frames Per Second. Furthermore, with the assistance of an inspection robot, the proposed dead hen detection model was deployed in the actual farming environments. Compared with methods proposed by other researchers, the proposed model is more suitable for complex actual farming environments and achieves higher detection accuracy, which can offer a reference for automated caged hen detection.

Knowledge Distillation for Enhanced Age and Gender Prediction Accuracy

In recent years, the ability to accurately predict age and gender from facial images has gained significant traction across various fields such as personalized marketing, human–computer interaction, and security surveillance. However, the high computational cost of the current models limits their practicality for real-time applications on resource-constrained devices. This study addressed this challenge by leveraging knowledge distillation to develop lightweight age and gender prediction models that maintain a high accuracy. We propose a knowledge distillation method using teacher bounds for the efficient learning of small models for age and gender. This method allows the student model to selectively receive the teacher model’s knowledge, preventing it from unconditionally learning from the teacher in challenging age/gender prediction tasks involving factors like illusions and makeup. Our experiments used MobileNetV3 and EfficientFormer as the student models and Vision Outlooker (VOLO)-D1 as the teacher model, resulting in substantial efficiency improvements. MobileNetV3-Small, one of the student models we experimented with, achieved a 94.27% reduction in parameters and a 99.17% reduction in Giga Floating Point Operations per Second (GFLOPs). Furthermore, the distilled MobileNetV3-Small model improved gender prediction accuracy from 88.11% to 90.78%. Our findings confirm that knowledge distillation can effectively enhance model performance across diverse demographic groups while ensuring efficiency for deployment on embedded devices. This research advances the development of practical, high-performance AI applications in resource-limited environments.

Lightweight CNN combined with knowledge distillation for the accurate determination of black tea fermentation degree

Black tea is the second most common type of tea in China. Fermentation is one of the most critical processes in its production, and it affects the quality of the finished product, whether it is insufficient or excessive. At present, the determination of black tea fermentation degree completely relies on artificial experience. It leads to inconsistent quality of black tea. To solve this problem, we use machine vision technology to distinguish the degree of fermentation of black tea based on images, this paper proposes a lightweight convolutional neural network (CNN) combined with knowledge distillation to discriminate the degree of fermentation of black tea. After comparing 12 kinds of CNN models, taking into account the size of the model and the performance of discrimination, as well as the selection principle of teacher models, Shufflenet_v2_x1.0 is selected as the student model, and Efficientnet_v2 is selected as the teacher model. Then, CrossEntropy Loss is replaced by Focal Loss. Finally, for Distillation Loss ratios of 0.6, 0.7, 0.8, 0.9, Soft Target Knowledge Distillation (ST), Masked Generative Distillation (MGD), Similarity-Preserving Knowledge Distillation (SPKD), and Attention Transfer (AT) four knowledge distillation methods are tested for their performance in distilling knowledge from the Shufflenet_v2_x1.0 model. The results show that the model discrimination performance after distillation is the best when the Distillation Loss ratio is 0.8 and the MGD method is used. This setup effectively improves the discrimination performance without increasing the number of parameters and computation volume. The model’s P, R and F1 values reach 0.9208, 0.9190 and 0.9192, respectively. It achieves precise discrimination of the fermentation degree of black tea. This meets the requirements of objective black tea fermentation judgment and provides technical support for the intelligent processing of black tea.

Multimodal Machine Translation Based on Enhanced Knowledge Distillation and Feature Fusion

Existing research on multimodal machine translation (MMT) has typically enhanced bilingual translation by introducing additional alignment visual information. However, picture form requirements in multimodal datasets pose important constraints on the development of MMT because this requires a form of alignment between image, source text, and target text. This limitation is especially compounded by the fact that the inference phase, when aligning images, is not directly available in a conventional neural machine translation (NMT) setup. Therefore, we propose an innovative MMT framework called the DSKP-MMT model, which supports machine translation by enhancing knowledge distillation and feature refinement methods in the absence of images. Our model first generates multimodal features from the source text. Then, the purified features are obtained through the multimodal feature generator and knowledge distillation module. The features generated through image feature enhancement are subsequently further purified. Finally, the image–text fusion features are generated as input in the transformer-based machine translation reasoning task. In the Multi30K dataset test, the DSKP-MMT model has achieved a BLEU of 40.42 and a METEOR of 58.15, showing its ability to improve translation effectiveness and facilitating utterance communication.

Vision-Language Models for Vision Tasks: A Survey.

Most visual recognition studies rely heavily on crowd-labelled data in deep neural networks (DNNs) training, and they usually train a DNN for each single visual recognition task, leading to a laborious and time-consuming visual recognition paradigm. To address the two challenges, Vision-Language Models (VLMs) have been intensively investigated recently, which learns rich vision-language correlation from web-scale image-text pairs that are almost infinitely available on the Internet and enables zero-shot predictions on various visual recognition tasks with a single VLM. This paper provides a systematic review of visual language models for various visual recognition tasks, including: (1) the background that introduces the development of visual recognition paradigms; (2) the foundations of VLM that summarize the widely-adopted network architectures, pre-training objectives, and downstream tasks; (3) the widely-adopted datasets in VLM pre-training and evaluations; (4) the review and categorization of existing VLM pre-training methods, VLM transfer learning methods, and VLM knowledge distillation methods; (5) the benchmarking, analysis and discussion of the reviewed methods; (6) several research challenges and potential research directions that could be pursued in the future VLM studies for visual recognition.

A Lightweight and High-Precision Passion Fruit YOLO Detection Model for Deployment in Embedded Devices.

In order to shorten detection times and improve average precision in embedded devices, a lightweight and high-accuracy model is proposed to detect passion fruit in complex environments (e.g., with backlighting, occlusion, overlap, sun, cloud, or rain). First, replacing the backbone network of YOLOv5 with a lightweight GhostNet model reduces the number of parameters and computational complexity while improving the detection speed. Second, a new feature branch is added to the backbone network and the feature fusion layer in the neck network is reconstructed to effectively combine the lower- and higher-level features, which improves the accuracy of the model while maintaining its lightweight nature. Finally, a knowledge distillation method is used to transfer knowledge from the more capable teacher model to the less capable student model, significantly improving the detection accuracy. The improved model is denoted as G-YOLO-NK. The average accuracy of the G-YOLO-NK network is 96.00%, which is 1.00% higher than that of the original YOLOv5s model. Furthermore, the model size is 7.14 MB, half that of the original model, and its real-time detection frame rate is 11.25 FPS when implemented on the Jetson Nano. The proposed model is found to outperform state-of-the-art models in terms of average precision and detection performance. The present work provides an effective model for real-time detection of passion fruit in complex orchard scenes, offering valuable technical support for the development of orchard picking robots and greatly improving the intelligence level of orchards.

Cross-modal knowledge distillation for continuous sign language recognition

Continuous Sign Language Recognition (CSLR) is a task which converts a sign language video into a gloss sequence. The existing deep learning based sign language recognition methods usually rely on large-scale training data and rich supervised information. However, current sign language datasets are limited, and they are only annotated at sentence-level rather than frame-level. Inadequate supervision of sign language data poses a serious challenge for sign language recognition, which may result in insufficient training of sign language recognition models. To address above problems, we propose a cross-modal knowledge distillation method for continuous sign language recognition, which contains two teacher models and one student model. One of the teacher models is the Sign2Text dialogue teacher model, which takes a sign language video and a dialogue sentence as input and outputs the sign language recognition result. The other teacher model is the Text2Gloss translation teacher model, which targets to translate a text sentence into a gloss sequence. Both teacher models can provide information-rich soft labels to assist the training of the student model, which is a general sign language recognition model. We conduct extensive experiments on multiple commonly used sign language datasets, i.e., PHOENIX 2014T, CSL-Daily and QSL, the results show that the proposed cross-modal knowledge distillation method can effectively improve the sign language recognition accuracy by transferring multi-modal information from teacher models to the student model. Code is available at https://github.com/glq-1992/cross-modal-knowledge-distillation_new.

Decoupled graph knowledge distillation: A general logits-based method for learning MLPs on graphs

While Graph Neural Networks (GNNs) have demonstrated their effectiveness in processing non-Euclidean structured data, the neighborhood fetching of GNNs is time-consuming and computationally intensive, making them difficult to deploy in low-latency industrial applications. To address the issue, a feasible solution is graph knowledge distillation (KD), which can learn high-performance student Multi-layer Perceptrons (MLPs) to replace GNNs by mimicking the superior output of teacher GNNs. However, state-of-the-art graph knowledge distillation methods are mainly based on distilling deep features from intermediate hidden layers, this leads to the significance of logit layer distillation being greatly overlooked. To provide a novel viewpoint for studying logits-based KD methods, we introduce the idea of decoupling into graph knowledge distillation. Specifically, we first reformulate the classical graph knowledge distillation loss into two parts, i.e., the target class graph distillation (TCGD) loss and the non-target class graph distillation (NCGD) loss. Next, we decouple the negative correlation between GNN’s prediction confidence and NCGD loss, as well as eliminate the fixed weight between TCGD and NCGD. We named this logits-based method Decoupled Graph Knowledge Distillation (DGKD). It can flexibly adjust the weights of TCGD and NCGD for different data samples, thereby improving the prediction accuracy of the student MLP. Extensive experiments conducted on public benchmark datasets show the effectiveness of our method. Additionally, DGKD can be incorporated into any existing graph knowledge distillation framework as a plug-and-play loss function, further improving distillation performance. The code is available at https://github.com/xsk160/DGKD.

Research on Isomorphic Task Transfer Algorithm Based on Knowledge Distillation in Multi-Agent Collaborative Systems.

In response to the increasing number of agents and changing task scenarios in multi-agent collaborative systems, existing collaborative strategies struggle to effectively adapt to new task scenarios. To address this challenge, this paper proposes a knowledge distillation method combined with a domain separation network (DSN-KD). This method leverages the well-performing policy network from a source task as the teacher model, utilizes a domain-separated neural network structure to correct the teacher model's outputs as supervision, and guides the learning of agents in new tasks. The proposed method does not require the pre-design or training of complex state-action mappings, thereby reducing the cost of transfer. Experimental results in scenarios such as UAV surveillance and UAV cooperative target occupation, robot cooperative box pushing, UAV cooperative target strike, and multi-agent cooperative resource recovery in a particle simulation environment demonstrate that the DSN-KD transfer method effectively enhances the learning speed of new task policies and improves the proximity of the policy model to the theoretically optimal policy in practical tasks.

Identification of internal voids in pavement based on improved knowledge distillation technology

Investigating methods for the detection of internal voids within road structures is a critical measure to ensure the safety and integrity of roadway operations. The purpose of this research is to investigate on the identification method of internal voids in pavement based on improved knowledge distillation technology. Ground penetrating radar data in three dimensions were extensively collected to capture the internal voids present within roadways, and this data was subsequently validated through in-situ verification. The echo characteristics of ground penetrating radar for areas with road voids were analyzed, and a dataset containing 1700 images of these internal voids was established. A YOLOv8 model improvement method was proposed, and a model for the detection of internal road voids was constructed based on the improved YOLO v8 framework. To further refine the model's performance, a knowledge distillation method based on multiple guidance from teacher assistants was developed. A stochastic learning approach was integrated, resulting in the establishment of a model optimized by this stochastic learning scheme for the identification of internal road voids. The results demonstrate that the presence of overfitting during the training phase of the void identification model can restrict its performance within a certain domain. The proposed stochastic learning-optimized, multi-teacher assistant guided knowledge distillation model, adeptly harnesses the performance benefits of both the teacher and assistant models by means of knowledge transfer, consequently achieving a significant improvement in the detection of internal road voids.

Shape-position perceptive fusion electronic skin with autonomous learning for gesture interaction

Wearable devices, such as data gloves and electronic skins, can perceive human instructions, behaviors and even emotions by tracking a hand's motion, with the help of knowledge learning. The shape or position single-mode sensor in such devices often lacks comprehensive information to perceive interactive gestures. Meanwhile, the limited computing power of wearable applications restricts the multimode fusion of different sensing data and the deployment of deep learning networks. We propose a perceptive fusion electronic skin (PFES) with a bioinspired hierarchical structure that utilizes the magnetization state of a magnetostrictive alloy film to be sensitive to external strain or magnetic field. Installed at the joints of a hand, the PFES realizes perception of curvature (joint shape) and magnetism (joint position) information by mapping corresponding signals to the two-directional continuous distribution such that the two edges represent the contributions of curvature radius and magnetic field, respectively. By autonomously selecting knowledge closer to the user's hand movement characteristics, the reinforced knowledge distillation method is developed to learn and compress a teacher model for rapid deployment on wearable devices. The PFES integrating the autonomous learning algorithm can fuse curvature-magnetism dual information, ultimately achieving human machine interaction with gesture recognition and haptic feedback for cross-space perception and manipulation.

An improved multi-scale and knowledge distillation method for efficient pedestrian detection in dense scenes

An improved multi-scale and knowledge distillation method for efficient pedestrian detection in dense scenes