Knowledge Distillation Approach Research Articles

Air Traffic Flow Prediction with Spatiotemporal Knowledge Distillation Network

Air Traffic Flow Prediction with Spatiotemporal Knowledge Distillation Network

MCAD: Multi-classification anomaly detection with relational knowledge distillation

AbstractWith the wide application of deep learning in anomaly detection (AD), industrial vision AD has achieved remarkable success. However, current AD usually focuses on anomaly localization and rarely investigates anomaly classification. Furthermore, anomaly classification is currently requested for quality management and anomaly reason analysis. Therefore, it is essential to classify anomalies while improving the accuracy of AD. This paper designs a novel multi-classification AD (MCAD) framework to achieve high-accuracy AD with an anomaly classification function. In detail, the proposal model based on relational knowledge distillation consists of two components. The first one employs a teacher–student AD model, utilizing a relational knowledge distillation approach to transfer the interrelationships of images. The teacher–student critical layer feature activation values are used in the knowledge transfer process to achieve anomaly detection. The second component realizes anomaly multi-classification using the lightweight convolutional neural network. Our proposal has achieved 98.95, 96.04, and 92.94% AUROC AD results on MNIST, FashionMNIST, and CIFAR10 datasets. Meanwhile, we earn 97.58 and 98.10% AUROC for AD and localization in the MVTecAD dataset. The average classification accuracy of anomaly classification has reached 76.37% in fifteen categories of the MVTec-AD dataset. In particular, the classification accuracy of the leather category has gained 95.24%. The results on the MVTec-AD dataset show that MCAD achieves excellent detection, localization, and classification results.

Xai-driven knowledge distillation of large language models for efficient deployment on low-resource devices

Large Language Models (LLMs) are characterized by their inherent memory inefficiency and compute-intensive nature, making them impractical to run on low-resource devices and hindering their applicability in edge AI contexts. To address this issue, Knowledge Distillation approaches have been adopted to transfer knowledge from a complex model, referred to as the teacher, to a more compact, computationally efficient one, known as the student. The aim is to retain the performance of the original model while substantially reducing computational requirements. However, traditional knowledge distillation methods may struggle to effectively transfer crucial explainable knowledge from an LLM teacher to the student, potentially leading to explanation inconsistencies and decreased performance. This paper presents DiXtill, a method based on a novel approach to distilling knowledge from LLMs into lightweight neural architectures. The main idea is to leverage local explanations provided by an eXplainable Artificial Intelligence (XAI) method to guide the cross-architecture distillation of a teacher LLM into a self-explainable student, specifically a bi-directional LSTM network.Experimental results show that our XAI-driven distillation method allows the teacher explanations to be effectively transferred to the student, resulting in better agreement compared to classical distillation methods,thus enhancing the student interpretability. Furthermore, it enables the student to achieve comparable performance to the teacher LLM while also delivering a significantly higher compression ratio and speedup compared to other techniques such as post-training quantization and pruning, which paves the way for more efficient and sustainable edge AI applications

Distilling Structural Knowledge for Platform-Aware Semantic Segmentation

Knowledge Distillation (KD) aims to distill the dark knowledge of a high-powered teacher network into a student network, which can improve the capacity of student network and has been successfully applied to semantic segmentation. However, the standard knowledge distillation approaches merely represent the supervisory signal of teacher network as the dark knowledge, while ignoring the impact of network architecture during distillation. In this paper, we found that the student network with a more similar architecture against the teacher network obtains more performance gain from distillation. Therefore, a more generalized paradigm for knowledge distillation is to distill both the soft-label and the structure of the teacher network. We propose a novel Structural Distillation (SD) method which introduces the structural similarity constraints into vanilla knowledge distillation. We leverage Neural Architecture Search technique to search optimal student structure for semantic segmentation from a well-designed search space, which mimics the given teacher both in terms of soft-label and network structure. Experiment results make clear that our proposed method outperforms both the NAS with conventional Knowledge Distillation and human-designed methods, and achieves sota performance on the Cityscapes dataset under various platform-aware latency constraints. Furthermore, the best architecture discovered on Cityscapes also transfers well to the PASCAL VOC2012 dataset.

From sensor fusion to knowledge distillation in collaborative LIBS and hyperspectral imaging for mineral identification

Multimodal spectral imaging offers a unique approach to the enhancement of the analytical capabilities of standalone spectroscopy techniques by combining information gathered from distinct sources. In this manuscript, we explore such opportunities by focusing on two well-known spectral imaging techniques, namely laser-induced breakdown spectroscopy, and hyperspectral imaging, and explore the opportunities of collaborative sensing for a case study involving mineral identification. In specific, the work builds upon two distinct approaches: a traditional sensor fusion, where we strive to increase the information gathered by including information from the two modalities; and a knowledge distillation approach, where the Laser Induced Breakdown spectroscopy is used as an autonomous supervisor for hyperspectral imaging. Our results show the potential of both approaches in enhancing the performance over a single modality sensing system, highlighting, in particular, the advantages of the knowledge distillation framework in maximizing the potential benefits of using multiple techniques to build more interpretable models and paving for industrial applications.

Neural Network-based Pattern Recognition in the Framework of Edge Computing

Neural network (NN) model has been widely used in pattern recognition (PR), speech recognition, image processing and other fields, but its application in edge computing (EC) environment faces performance and energy consumption problems. This article first introduced the basic structure and training process of NN, including backpropagation algorithms. Then, this article presented a NN modeling approach based on EC, including NN model compression, distributed NN model and knowledge distillation approach. Finally, this article implemented a PR model for the MNIST (Mixed National Institute of Standards and Technology database) dataset and analyzed the experimental results. The experimental outcomes indicated that the presented approach can significantly enhance the performance of the NN model in the EC environment, while ensuring a high recognition accuracy. The NN modeling approach based on EC can reduce the amount of computation and storage of the NN, thus improving the operating efficiency of the NN in the EC environment by 6%-12%. The NN modeling approach based on EC can optimize the performance and efficiency of the NN model in the EC environment, and provide new ideas and approaches for the application of NN in the EC environment.

Sunshine to Rainstorm: Cross-Weather Knowledge Distillation for Robust 3D Object Detection

LiDAR-based 3D object detection models inevitably struggle under rainy conditions due to the degraded and noisy scanning signals. Previous research has attempted to address this by simulating the noise from rain to improve the robustness of detection models. However, significant disparities exist between simulated and actual rain-impacted data points. In this work, we propose a novel rain simulation method, termed DRET, that unifies Dynamics and Rainy Environment Theory to provide a cost-effective means of expanding the available realistic rain data for 3D detection training. Furthermore, we present a Sunny-to-Rainy Knowledge Distillation (SRKD) approach to enhance 3D detection under rainy conditions. Extensive experiments on the Waymo-Open-Dataset show that, when combined with the state-of-the-art DSVT model and other classical 3D detectors, our proposed framework demonstrates significant detection accuracy improvements, without losing efficiency. Remarkably, our framework also improves detection capabilities under sunny conditions, therefore offering a robust solution for 3D detection regardless of whether the weather is rainy or sunny.

Cytopathology image analysis method based on high-resolution medical representation learning in medical decision-making system

Artificial intelligence has made substantial progress in many medical application scenarios. The quantity and complexity of pathology images are enormous, but conventional visual screening techniques are labor-intensive, time-consuming, and subject to some degree of subjectivity. Complex pathological data can be converted into mineable image features using artificial intelligence image analysis technology, enabling medical professionals to quickly and quantitatively identify regions of interest and extract information about cellular tissue. In this study, we designed a medical information assistance system for segmenting pathology images and quantifying statistical results, including data enhancement, cell nucleus segmentation, model tumor, and quantitative analysis. In cell nucleus segmentation, to address the problem of uneven healthcare resources, we designed a high-precision teacher model (HRMED_T) and a lightweight student model (HRMED_S). The HRMED_T model is based on visual Transformer and high-resolution representation learning. It achieves accurate segmentation by parallel low-resolution convolution and high-scaled image iterative fusion, while also maintaining the high-resolution representation. The HRMED_S model is based on the Channel-wise Knowledge Distillation approach to simplify the structure, achieve faster convergence, and refine the segmentation results by using conditional random fields instead of fully connected structures. The experimental results show that our system has better performance than other methods. The Intersection over the Union (IoU) of HRMED_T model reaches 0.756. The IoU of HRMED_S model also reaches 0.710 and params is only 3.99 M.

TraKDis: A Transformer-Based Knowledge Distillation Approach for Visual Reinforcement Learning With Application to Cloth Manipulation

TraKDis: A Transformer-Based Knowledge Distillation Approach for Visual Reinforcement Learning With Application to Cloth Manipulation

Tolerant Self-Distillation for image classification

Deep neural networks tend to suffer from the overfitting issue when the training data are not enough. In this paper, we introduce two metrics from the intra-class distribution of correct-predicted and incorrect-predicted samples to provide a new perspective on the overfitting issue. Based on it, we propose a knowledge distillation approach without pretraining a teacher model in advance named Tolerant Self-Distillation (TSD) for alleviating the overfitting issue. It introduces an online updating memory and selectively stores the class predictions of the samples from the past iterations, making it possible to distill knowledge across the iterations. Specifically, the class predictions stored in the memory bank serve as the soft labels for supervising the samples from the same class for the current iteration in a reverse way, i.e. the correct-predicted samples are supervised with the incorrect predictions while the incorrect-predicted samples are supervised with the correct predictions. Consequently, the premature convergence issue caused by the over-confident samples would be mitigated, which helps the model to converge to a better local optimum. Extensive experimental results on several image classification benchmarks, including small-scale, large-scale, and fine-grained datasets, demonstrate the superiority of the proposed TSD.

Detection of COVID-19, pneumonia, and tuberculosis from radiographs using AI-driven knowledge distillation

Chest radiography is an essential diagnostic tool for respiratory diseases such as COVID-19, pneumonia, and tuberculosis because it accurately depicts the structures of the chest. However, accurate detection of these diseases from radiographs is a complex task that requires the availability of medical imaging equipment and trained personnel. Conventional deep learning models offer a viable automated solution for this task. However, the high complexity of these models often poses a significant obstacle to their practical deployment within automated medical applications, including mobile apps, web apps, and cloud-based platforms. This study addresses and resolves this dilemma by reducing the complexity of neural networks using knowledge distillation techniques (KDT). The proposed technique trains a neural network on an extensive collection of chest X-ray images and propagates the knowledge to a smaller network capable of real-time detection. To create a comprehensive dataset, we have integrated three popular chest radiograph datasets with chest radiographs for COVID-19, pneumonia, and tuberculosis. Our experiments show that this knowledge distillation approach outperforms conventional deep learning methods in terms of computational complexity and performance for real-time respiratory disease detection. Specifically, our system achieves an impressive average accuracy of 0.97, precision of 0.94, and recall of 0.97.

Hybrid mix-up contrastive knowledge distillation

Knowledge distillation (KD) aims to build a lightweight deep neural network model under the guidance of a large-scale teacher model for model simplicity. Despite improved model efficiency through the KD technique, the performance gap between a teacher model and the trained student model remains significant. This is because the knowledge of the teacher model is not effectively transferred to the student model since the mapping landscape of the large-scale teacher model is not fully explored. To tackle this research gap, we propose a novel Hybrid Mix-up Contrastive Knowledge Distillation (HMCKD) approach, which facilitates a thorough and reliable mapping solution space exploration in order to significantly improve the performance of the student model. Specifically, we design a hybrid mixing strategy, including image-level mixing and feature-level mixing, to form a smoother mapping landscape as a means to provide a stronger guidance in order to embed its richer dark knowledge from a teacher model to its student model. Additionally, we apply two other strategies, including contrastive learning and top-k guided selection, in order to ensure more effective knowledge transferability from the teacher model. Extensive experiments have proved that our proposed HMCKD approach outperforms state-of-the-art knowledge distillation methods when tested on 6 publicly available datasets, such as CIFAR-100, CIFAR-100-C, STL-10, SVHN, TinyImageNet, and ImageNet. Particularly on CIFAR-100 dataset, the average accuracy of students using HMCKD increased by 1.47%. Further, both the visualization results and similarity quantifications have confirmed the narrowed knowledge gap between the teacher and student models. Our source code is available at https://github.com/lambett/HMCKD.

Imbalance Mitigation for Continual Learning via Knowledge Decoupling and Dual Enhanced Contrastive Learning.

Continual learning (CL) aims at studying how to learn new knowledge continuously from data streams without catastrophically forgetting the previous knowledge. One of the key problems is catastrophic forgetting, that is, the performance of the model on previous tasks declines significantly after learning the subsequent task. Several studies addressed it by replaying samples stored in the buffer when training new tasks. However, the data imbalance between old and new task samples results in two serious problems: information suppression and weak feature discriminability. The former refers to the information in the sufficient new task samples suppressing that in the old task samples, which is harmful to maintaining the knowledge since the biased output worsens the consistency of the same sample's output at different moments. The latter refers to the feature representation being biased to the new task, which lacks discrimination to distinguish both old and new tasks. To this end, we build an imbalance mitigation for CL (IMCL) framework that incorporates a decoupled knowledge distillation (DKD) approach and a dual enhanced contrastive learning (DECL) approach to tackle both problems. Specifically, the DKD approach alleviates the suppression of the new task on the old tasks by decoupling the model output probability during the replay stage, which better maintains the knowledge of old tasks. The DECL approach enhances both low-and high-level features and fuses the enhanced features to construct contrastive loss to effectively distinguish different tasks. Extensive experiments on three popular datasets show that our method achieves promising performance under task incremental learning (Task-IL), class incremental learning (Class-IL), and domain incremental learning (Domain-IL) settings.

GC2: Generalizable Continual Classification of Medical Images.

Deep learning models have achieved remarkable success in medical image classification. These models are typically trained once on the available annotated images and thus lack the ability of continually learning new tasks (i.e., new classes or data distributions) due to the problem of catastrophic forgetting. Recently, there has been more interest in designing continual learning methods to learn different tasks presented sequentially over time while preserving previously acquired knowledge. However, these methods focus mainly on preventing catastrophic forgetting and are tested under a closed-world assumption; i.e., assuming the test data is drawn from the same distribution as the training data. In this work, we advance the state-of-the-art in continual learning by proposing GC2 for medical image classification, which learns a sequence of tasks while simultaneously enhancing its out-of-distribution robustness. To alleviate forgetting, GC2 employs a gradual culpability-based network pruning to identify an optimal subnetwork for each task. To improve generalization, GC2 incorporates adversarial image augmentation and knowledge distillation approaches for learning generalized and robust representations for each subnetwork. Our extensive experiments on multiple benchmarks in a task-agnostic inference demonstrate that GC2 significantly outperforms baselines and other continual learning methods in reducing forgetting and enhancing generalization. Our code is publicly available at the following link: https://github.com/ nourhanb/TMI2024-GC2.

Remaining useful life prediction across machines using multi-source adversarial online knowledge distillation

Deep transfer learning has been extensively developed in the remaining useful life prediction of rolling bearings because it can decrease the dependence on massive labeled data and robustly extract adaptive features. Most existing studies perform remaining useful life predictions across different working conditions within the same machine. Therefore, these methods are unsuitable for data with severe distribution drifts. Only a few studies investigate the remaining useful life prediction across different devices, but their success depends on the prerequisite that a large amount of labeled data exists. In brief, the models' applicabilities and prediction performances are significantly constrained toward industrial scenarios. To address these drawbacks, a multi-source adversarial online knowledge distillation approach is proposed for rolling bearing remaining useful life prediction across machines. A dual knowledge transfer mechanism, including multi-level domain adaptation and online knowledge distillation with dynamic weighting, is designed in the ensemble learning architecture. The proposed approach allows the more comprehensive extraction and transition of prognostic knowledge of multiple working conditions across different machines, thereby improving the performance of remaining useful life prediction. Experiments on two open-access rolling bearing datasets demonstrate the effectiveness and superiority of the proposed approach in terms of prediction accuracy.

Designing an improved deep learning-based model for COVID-19 recognition in chest X-ray images: a knowledge distillation approach

Designing an improved deep learning-based model for COVID-19 recognition in chest X-ray images: a knowledge distillation approach

Soft Hybrid Knowledge Distillation against deep neural networks

Traditional knowledge distillation approaches are typically designed for specific tasks, as they primarily distilling deep features from intermediate layers of a neural network, generally with ingeniously designed knowledge representations, which increase the difficulty of model development and interpretation. In contrast, we empirically show that a soft logits knowledge distillation is enough to significantly narrow the teacher–student model performance gap. In this paper, the Soft Hybrid Knowledge Distillation (SHKD) is proposed to generate representative features from the input and output location of models, and applicable to various teacher and student architectures. Specifically, the additional mixup task is first applied to the same batch of training images, which enriches the high-order feature information transferred by the model. And then, a top-k guided specification filter is added to the teacher model, to reduce the bias induced throughout the knowledge transfer process. Finally, the SHKD utilizes the same and different teacher–student model pairs of the deep neural network, which can be easily adjusted to the computational resources available in the current world. Extensive experiments are conducted to examine the efficiency, and show that the SHKD outperforms state-of-the-art knowledge distillation methods. With the SHKD, the high performance of models will be maintained, and the gap between the teacher model and the student model can be reduced, making the construction of deep neural networks more convenient to the unknown scenarios. The source code is available at https://github.com/lambett/SHKD.

Mitigating misalignment in MRI-to-CT synthesis for improved synthetic CT generation: an iterative refinement and knowledge distillation approach

Objective. Deep learning has shown promise in generating synthetic CT (sCT) from magnetic resonance imaging (MRI). However, the misalignment between MRIs and CTs has not been adequately addressed, leading to reduced prediction accuracy and potential harm to patients due to the generative adversarial network (GAN)hallucination phenomenon. This work proposes a novel approach to mitigate misalignment and improve sCT generation. Approach. Our approach has two stages: iterative refinement and knowledge distillation. First, we iteratively refine registration and synthesis by leveraging their complementary nature. In each iteration, we register CT to the sCT from the previous iteration, generating a more aligned deformed CT (dCT). We train a new model on the refined 〈dCT, MRI〉 pairs to enhance synthesis. Second, we distill knowledge by creating a target CT (tCT) that combines sCT and dCT images from the previous iterations. This further improves alignment beyond the individual sCT and dCT images. We train a new model with the 〈tCT, MRI〉 pairs to transfer insights from multiple models into this final knowledgeable model. Main results. Our method outperformed conditional GANs on 48 head and neck cancer patients. It reduced hallucinations and improved accuracy in geometry (3% ↑ Dice), intensity (16.7% ↓ MAE), and dosimetry (1% ↑ γ 3%3mm). It also achieved <1% relative dose difference for specific dose volume histogram points. Significance. This pioneering approach for addressing misalignment shows promising performance in MRI-to-CT synthesis for MRI-only planning. It could be applied to other modalities like cone beam computed tomography and tasks such as organ contouring.

Self-supervised knowledge distillation in counterfactual learning for VQA

As a popular cross-modal reasoning task, Visual Question Answering (VQA) has achieved great progress in recent years. However, the issue of language bias has always affected the reliability of VQA models. To address this problem, counterfactual learning methods are proposed to learn more robust features to mitigate the bias problem. However, current counterfactual learning approaches mainly focus on generating synthesized samples and assigning answers to them, neglecting the relationship between factual and original data, which hinders robust feature learning for effective reasoning. To overcome this limitation, we propose a Self-supervised Knowledge Distillation approach in Counterfactual Learning for VQA, dubbed as VQA-SkdCL, which utilizes a self-supervised constraint to make good use of the hidden knowledge in the factual samples, enhancing the robustness of VQA models. We demonstrate the effectiveness of the proposed approach on VQA v2, VQA-CP v1, and VQA-CP v2 datasets and our approach achieves excellent performance.

Forest Fire Object Detection Analysis Based on Knowledge Distillation

This paper investigates the application of the YOLOv7 object detection model combined with knowledge distillation techniques in forest fire detection. As an advanced object detection model, YOLOv7 boasts efficient real-time detection capabilities. However, its performance may be constrained in resource-limited environments. To address this challenge, this research proposes a novel approach: considering that deep neural networks undergo multi-layer mapping from the input to the output space, we define the knowledge propagation between layers by evaluating the dot product of features extracted from two different layers. To this end, we utilize the Flow of Solution Procedure (FSP) matrix based on the Gram matrix and redesign the distillation loss using the Pearson correlation coefficient, presenting a new knowledge distillation method termed ILKDG (Intermediate Layer Knowledge Distillation with Gram Matrix-based Feature Flow). Compared with the classical knowledge distillation algorithm, KD, ILKDG achieved a significant performance improvement on a self-created forest fire detection dataset. Specifically, without altering the student network’s parameters or network layers, mAP@0.5 improved by 2.9%, and mAP@0.5:0.95 increased by 2.7%. These results indicate that the proposed ILKDG method effectively enhances the accuracy and performance of forest fire detection without introducing additional parameters. The ILKDG method, based on the Gram matrix and Pearson correlation coefficient, presents a novel knowledge distillation approach, providing a fresh avenue for future research. Researchers can further optimize and refine this method to achieve superior results in fire detection.