Knowledge Distillation Research Articles

Optimizing Federated Learning Efficiency: A Comparative Analysis of Model Compression Techniques for Communication Reduction

Abstract. The rapidly growing number of connected devices continuously produces massive amounts of heterogeneous data, posing challenges to data privacy and security when leveraging them to train the high-quality big model. Federated learning, which enables numerous users to train a global model without cooperatively exchanging data, has emerged as a viable alternative. Though achieving significant progress, existing federated learning methods still struggle with large communication volumes, especially when the local devices only have limited computing and communication capabilities. To alleviate the efficiency issue, this paper compares the effect of three compression methods in promoting the training of federated learning models, including pruning, quantization, and knowledge distillation. The findings reveal that these methods reduce resource consumption while maintaining high model performance. Combining the pruning, quantization, and knowledge distillation technology through sequential application and parameter aggregation helps balance the model size and performance. Our cascading lightweight strategy, which preserves each method's unique edge while promoting deeper collaboration between them, has been shown beneficial through extensive testing.

Fault diagnosis method for elevator carriages based on temporal generative federated distillation

Addressing the ‘data silo’ issue among different elevator operating units and the temporal correlations in elevator vibration signals, a novel small-sample fault diagnosis method for elevator carriages based on temporal generative federated distillation is proposed. This method incorporates a temporal generative adversarial network into Federated Distillation via Generative Learning (FedGen). FedGen combines federated learning, knowledge distillation, and generative models to enhance model aggregation efficiency while mitigating data heterogeneity. However, the original generative model struggles to maintain dynamic correlations between signals when extracting temporal features. Therefore, an improved Time Series Generative Adversarial Networks (TimeGAN) model is introduced, substituting the initial logarithmic loss function with a least squares error function, thereby enhancing training stability and data quality. This approach eliminates the need for proxy datasets in knowledge distillation, avoiding the loss of temporal information during central server feature extraction. Simulation results demonstrate that this method enables data sharing while protecting data privacy, and enhances model generalization capabilities.

Foreground separation knowledge distillation for object detection

In recent years, deep learning models have become predominant methods for computer vision tasks, but the large computation and storage requirements of many models make them challenging to deploy on devices with limited resources. Knowledge distillation (KD) is a widely used approach for model compression. However, when applied in the object detection problems, the existing KD methods either directly applies the feature map or simply separate the foreground from the background by using a binary mask, aligning the attention between the teacher and the student models. Unfortunately, these methods either completely overlook or fail to thoroughly eliminate noise, resulting in unsatisfactory model accuracy for student models. To address this issue, we propose a foreground separation distillation (FSD) method in this paper. The FSD method enables student models to distinguish between foreground and background using Gaussian heatmaps, reducing irrelevant information in the learning process. Additionally, FSD also extracts the channel feature by converting the spatial feature maps into probabilistic forms to fully utilize the knowledge in each channel of a well-trained teacher. Experimental results demonstrate that the YOLOX detector enhanced with our distillation method achieved superior performance on both the fall detection and the VOC2007 datasets. For example, YOLOX with FSD achieved 73.1% mean average precision (mAP) on the Fall Detection dataset, which is 1.6% higher than the baseline. The code of FSD is accessible via https://doi.org/10.5281/zenodo.13829676.

Pre-Impact Fall Detection for E-Scooter Riding Using an IMU: Threshold-Based, Supervised, and Unsupervised Approaches

Pre-impact fall detection during e-scooter riding is essential for rider safety. Both threshold-based and deep learning algorithms (supervised and unsupervised models) were developed in this study. Twenty participants performed normal driving maneuvers such as straight driving, speed bumps, clockwise roundabouts, and counterclockwise roundabouts, along with falls (abnormal driving maneuvers). A 6-axis IMU sensor (Xsens DOT, The Netherlands) was positioned at the T7 location to record data at 60 Hz. The approaches included threshold-based, supervised learning, and unsupervised learning models The threshold-based approach yielded an accuracy of 98.86% with an F1 score of 0.99, while the supervised model had a slightly lower performance, reaching 86.29% accuracy and an F1 score of 0.56. The unsupervised knowledge distillation model achieved 98.86% accuracy, an F1 score of 0.99, and a memory size of only 46 kB. All models demonstrated lead times of more than 250 ms, sufficient for airbag deployment.

Toward Cross-Lingual Social Event Detection with Hybrid Knowledge Distillation

Recently published graph neural networks (GNNs) show promising performance at social event detection tasks. However, most studies are oriented toward monolingual data in languages with abundant training samples. This has left the common lesser-spoken languages relatively unexplored. Thus, in this work, we present a GNN-based framework that integrates cross-lingual word embeddings into the process of graph knowledge distillation for detecting events in low-resource language data streams. To achieve this, a novel cross-lingual knowledge distillation framework, called CLKD, exploits prior knowledge learned from similar threads in English to make up for the paucity of annotated data. Specifically, to extract sufficient useful knowledge, we propose a hybrid distillation method that consists of both feature-wise and relation-wise information. To transfer both kinds of knowledge in an effective way, we add a cross-lingual module in the feature-wise distillation to eliminate the language gap and selectively choose beneficial relations in the relation-wise distillation to avoid distraction caused by teachers’ misjudgments. Our proposed CLKD framework also adopts different configurations to suit both offline and online situations. Experiments on real-world datasets show that the framework is highly effective at detection in languages where training samples are scarce.

Data free knowledge distillation with feature synthesis and spatial consistency for image analysis.

Privacy and security concerns restrict access to original training datasets, posing significant challenges for model compression. Data-Free Knowledge Distillation (DFKD) emerges as a solution, aiming to transfer knowledge from teacher to student networks without accessing original data. Existing DFKD methods struggle to generate high-quality synthetic samples that capture the complexities of real-world data, leading to suboptimal knowledge transfer. Moreover, these approaches often fail to preserve the spatial attributes of the teacher network, resulting in shortcut learning and limited generalization.To address these issues, a novel DFKD strategy is proposed with three innovations: (1) an enhanced DCGAN generator with an attention module for synthesizing samples with improved micro-discriminative features; (2) a Multi-Scale Spatial Activation Region Consistency (MSARC) mechanism to accurately replicate the teacher's spatial attributes; and (3) an adversarial learning framework that creates a dynamic competitive environment between the generative and distillation phases. Rigorous evaluation of the method on several benchmark datasets, including CIFAR-10, CIFAR-100, Tiny-ImageNet, and medical imaging datasets such as PathMNIST, BloodMNIST, and PneumoniaMNIST, demonstrates superior performance compared to existing DFKD methods. Specifically, on CIFAR-100, the student network attains an accuracy of 77.85%, surpassing previous methods like CMI and SpaceshipNet. On BloodMNIST, the method achieves an accuracy of 80.50%, outperforming the next best method by over 5%.

MonoFG: Monocular 3D Object Detection with Knowledge Distillation for Human-Centric Autonomous Driving Systems

Monocular 3D object detection is essential for identifying objects in road images, thus offering valuable environmental perception data that are crucial for human-centric autonomous driving systems. However, due to the inherent limitations of camera imaging, obtaining precise depth information from images alone is challenging, which hampers the accuracy of within-scene object localization. In this paper, we introduce a monocular 3D object detection method called MonoFG that uses knowledge distillation with separated foreground and background components to improve the accuracy of object localization. First, detached foreground and background distillation processes can strategically leverage the distinct positional information acquired from each location to optimize the produced global distillation effects. This step serves as the foundation for the subsequent feature and response distillation process, which focuses on the distilled foreground and background rather than isolated object distillation. Second, triple attention mechanism-based feature distillation intensifies the feature imitation and feature representation capabilities of the student network. Spatial and channel attention mechanisms encourage the student network to capture crucial pixels and channels from the teacher network, whereas a self-attention mechanism globally transfers the learned relationships between pixels. Third, localization error-based response distillation facilitates a clearer transfer of positional information from the teacher network to the student network. Only when the positioning ability of the teacher network exceeds that of the student network can knowledge be comprehensively distilled across both the foreground and background. Therefore, the distillation process is constrained to specific content, which is delineated by positioning errors that serve as the boundaries. Finally, experiments conducted on the KITTI benchmark dataset demonstrate that our method outperforms many well-known baseline methods in several representative evaluation tasks (e.g., 3D object detection and bird's-eye view (BEV) detection) involving human-centric autonomous driving systems.

Multi-scale contextual learning for medical image segmentation via dual distillation.

Recently, many studies have explored fusing features extracted from Convolutional neural networks (CNNs) and transformers to integrate multi-scale representations for better performance in medical image segmentation tasks. Although these hybrid models have achieved better results than previous CNN-based and transformer-based methods, they suffer from high computation and spacecomplexities. The purpose of this research is to address the prohibitive computation and space complexities of hybrid models, which limit their application in clinical practice where computational resources are usuallyconstrained. We propose a novel model equipped with a dual distillation scheme to sufficiently harness the complementary advantages of CNNs and transformers without compromising model efficiency. We further propose a multi-scale prior-knowledge distillation (MPD) module to effectively distill multi-scale knowledge from features extracted from transformers. In addition, to cooperate with the knowledge distillation scheme, we also propose an efficient and robust Selective Fusion module in the studentnetwork. We extensively evaluate the proposed model against fourteen different network frameworks on two representative datasets: SipakMed and ISIC 2017. In the SipakMed dataset, 3037 Pap smear images are used for training and 1012 for testing. In the ISIC 2017 dataset, 2000 dermoscopic images are used for training, 150 for validation, and 600 for testing. Experimental results demonstrate that our method not only surpasses existing methods by a considerable margin with respect to the evaluation metrics of mean Intersection over Union, mean Dice coefficient, mean average symmetric surface distance, but also requires fewer computational resources in terms of model parameters and floating-point operations persecond. Comprehensive comparisons in terms of segmentation accuracy and computational complexity unequivocally confirm that our method effectively and efficiently integrates the advantages of both CNNs and transformers, showing its suitability and significance for clinicalapplications.

KD-Crowd: a knowledge distillation framework for learning from crowds

KD-Crowd: a knowledge distillation framework for learning from crowds

Lightweight defect detection algorithm of tunnel lining based on knowledge distillation

Due to the influence of construction quality, engineering geology and hydrological environment, defects such as dehollowing and insufficient compaction can occur in tunnels. Aiming at the problems of complex detection model, poor real-time performance and low accuracy of the current tunnel lining defect detection methods, the study proposes a lightweight defect detection algorithm of tunnel lining based on knowledge distillation. Firstly, a high-precision teacher model based on yolov5s was constructed by constructing a C3CSFM module that combines residual structure and attention mechanism, a MDFPN network structure with multi-scale feature fusion and a reweighted RWNMS re-screening mechanism. Secondly, in the distillation process, the feature and output dimension results are fused to improve the detection accuracy, and the mask feature relationship is learned in the space and channel dimension to improve the real-time detection. Tests on the tunnel lining radar defect image dataset showed that the number of parameters of the improved model was reduced from 16.03 MB to 3.20 MB, a reduction of 80%, and the average accuracy was improved from 83.4 to 86.5%, an increase of 3.1%. On the basis of maintaining the structure and detection performance of the model, the lightweight degree of the model is greatly improved, and the high-precision and real-time detection of tunnel lining defects is realized.

Distilling Knowledge in Machine Translation of Agglutinative Languages with Backward and Morphological Decoders

Agglutinative languages often have morphologically complex words(MCWs) composed of multiple morphemes arranged in a hierarchical structure, posing significant challenges in translation tasks. We present a novel Knowledge Distillation approach tailored for improving the translation of such languages. Our method involves an encoder, a forward decoder, and two auxiliary decoders: a backward decoder and a morphological decoder. The forward decoder generates target morphemes autoregressively and is augmented by distilling knowledge from the auxiliary decoders. The backward decoder incorporates future context, while the morphological decoder integrates target-side morphological information. We have also designed a reliability estimation method to selectively distill only the reliable knowledge from these auxiliary decoders. Our approach relies on morphological word segmentation. We show that the word segmentation method based on unsupervised morphology learning outperforms the commonly used Byte Pair Encoding method on highly agglutinative languages in translation tasks. Our experiments conducted on English-Tamil, English-Manipuri, and English-Marathi datasets show that our proposed approach achieves significant improvements over strong Transformer-based NMT baselines.

SCARF: Scalable Continual Learning Framework for Memory‐efficient Multiple Neural Radiance Fields

AbstractThis paper introduces a novel continual learning framework for synthesising novel views of multiple scenes, learning multiple 3D scenes incrementally, and updating the network parameters only with the training data of the upcoming new scene. We build on Neural Radiance Fields (NeRF), which uses multi‐layer perceptron to model the density and radiance field of a scene as the implicit function. While NeRF and its extensions have shown a powerful capability of rendering photo‐realistic novel views in a single 3D scene, managing these growing 3D NeRF assets efficiently is a new scientific problem. Very few works focus on the efficient representation or continuous learning capability of multiple scenes, which is crucial for the practical applications of NeRF. To achieve these goals, our key idea is to represent multiple scenes as the linear combination of a cross‐scene weight matrix and a set of scene‐specific weight matrices generated from a global parameter generator. Furthermore, we propose an uncertain surface knowledge distillation strategy to transfer the radiance field knowledge of previous scenes to the new model. Representing multiple 3D scenes with such weight matrices significantly reduces memory requirements. At the same time, the uncertain surface distillation strategy greatly overcomes the catastrophic forgetting problem and maintains the photo‐realistic rendering quality of previous scenes. Experiments show that the proposed approach achieves state‐of‐the‐art rendering quality of continual learning NeRF on NeRF‐Synthetic, LLFF, and TanksAndTemples datasets while preserving extra low storage cost.

Non-small cell lung cancer detection through knowledge distillation approach with teaching assistant.

Non-small cell lung cancer (NSCLC) exhibits a comparatively slower rate of metastasis in contrast to small cell lung cancer, contributing to approximately 85% of the global patient population. In this work, leveraging CT scan images, we deploy a knowledge distillation technique within teaching assistant (TA) and student frameworks for NSCLC classification. We employed various deep learning models, CNN, VGG19, ResNet152v2, Swin, CCT, and ViT, and assigned roles as teacher, teaching assistant and student. Evaluation underscores exceptional model performance in performance metrics achieved via cost-sensitive learning and precise hyperparameter (alpha and temperature) fine-tuning, highlighting the model's efficiency in lung cancer tumor prediction and classification. The applied TA (ResNet152) and student (CNN) models achieved 90.99% and 94.53% test accuracies, respectively, with optimal hyperparameters (alpha = 0.7 and temperature = 7). The implementation of the TA framework improves the overall performance of the student model. After obtaining Shapley values, explainable AI is applied with a partition explainer to check each class's contribution, further enhancing the transparency of the implemented deep learning techniques. Finally, a web application designed to make it user-friendly and classify lung types in recently captured images. The execution of the three-stage knowledge distillation technique proved efficient with significantly reduced trainable parameters and training time applicable for memory-constrained edge devices.

Forget to Learn (F2L): Circumventing plasticity–stability trade-off in continuous unsupervised domain adaptation

In continuous unsupervised domain adaptation (CUDA), deep learning models struggle with the stability-plasticity trade-off—where the model must forget old knowledge to acquire new one. This paper introduces the “Forget to Learn” (F2L), a novel framework that circumvents such a trade-off. In contrast to state-of-the-art methods that aim to balance the two conflicting objectives, stability and plasticity, F2L utilizes active forgetting and knowledge distillation to circumvent the conflict’s root causes. In F2L, dual-encoders are trained, where the first encoder – the ‘Specialist’ – is designed to actively forget, thereby boosting adaptability (i.e., plasticity) and generating high-accuracy pseudo labels on the new domains. Such pseudo labels are then used to transfer/accumulate the specialist knowledge to the second encoder—the ‘Generalist’ through conflict-free knowledge distillation. Empirical and ablation studies confirmed F2L’s superiority on different datasets and against different SOTAs. Furthermore, F2L minimizes the need for hyperparameter tuning, enhances computational and sample efficiency, and excels in problems with long domain sequences—key advantages for practical systems constrained by hardware limitations.

Intelligent Fault Diagnosis Method Based on Neural Network Compression for Rolling Bearings

Rolling bearings are often exposed to high speeds and pressures, leading to the symmetry in their rotating structure being disrupted, which can lead to serious failures. Intelligent rolling bearing fault diagnosis is a critical part of ensuring operation of machinery, and it has been facilitated by the growing popularity of convolutional neural networks (CNNs). The outstanding performance of fault diagnosis CNNs results from complex and redundant network structures and parameters, resulting in huge storage and computational requirements, which makes it challenging to implement these models in resource-limited industrial devices. This study aims to address this problem by proposing a comprehensive compression method for CNNs that is applied to intelligent fault diagnosis. It involves several different compression methods, including tensor train decomposition, parameter quantization, and knowledge distillation for deep network compression. This results in a significant decrease in redundancy and speeding up the training of CNN models. Firstly, tensor train decomposition is applied to reduce redundant connections in both convolutional and fully connected layers. The next step is to perform parameter quantization to minimize the bits needed for parameter representation and storage. Finally, knowledge distillation is used to restore accuracy to the compressed model. The effectiveness of the proposed approach is confirmed by an experiment and ablation study with different models on several datasets. The results show that it can significantly reduce redundant information and floating-point operations with little degradation in accuracy. Notably, on the CWRU dataset, with about 60% parameter reduction, there is no degradation in our model’s accuracy. The proposed approach is a new attempt at the intelligent fault diagnosis of rolling bearings in industrial equipment.

Improving LIBS-based mineral identification with Raman imaging and spectral knowledge distillation

Combining data from different sensing modalities has been a promising research topic for building better and more reliable data-driven models. In particular, it is known that multimodal spectral imaging can improve the analytical capabilities of standalone spectroscopy techniques through fusion, hyphenation, or knowledge distillation techniques. In this manuscript, we focus on the latter, exploring how one can increase the performance of a Laser-induced Breakdown Spectroscopy system for mineral classification problems using additional spectral imaging techniques. Specifically, focusing on a scenario where Raman spectroscopy delivers accurate mineral classification performance, we show how to deploy a knowledge distillation pipeline where Raman spectroscopy may act as an autonomous supervisor for LIBS. For a case study concerning a challenging Li-bearing mineral identification of spodumene and petalite, our results demonstrate the advantages of this method in improving the performance of a single-technique system. LIBS trained with labels obtained by Raman presents an enhanced classification performance. Furthermore, leveraging the interpretability of the model deployed, the workflow opens opportunities for the deployment of assisted feature discovery pipelines, which may impact future academic and industrial applications.

Adaptive class token knowledge distillation for efficient vision transformer

The Vision Transformer (ViT) outperforms Convolutional Neural Networks (CNNs) but at the cost of significantly higher computational demands. Knowledge Distillation (KD) has shown promise in compressing complex networks by transferring knowledge from a large pre-trained model to a smaller one. However, current KD methods for ViT often rely on CNNs as teachers or neglect the importance of class token ([CLS]) information, resulting in ineffective distillation of ViT’s unique knowledge. In this paper, we propose Adaptive Class token Knowledge Distillation ([CLS]-KD), which fully exploits information from the class token and patches in ViT. For class embedding (CLS) distillation, the intermediate CLS of the student model is aligned with the corresponding CLS of the teacher model through a projector. Furthermore, we introduce CLS-patch attention map distillation, where an attention map between the CLS and patch embeddings is generated and matched at each layer. This empowers the student model to learn how to dynamically extract patch embedding information into the CLS under teacher guidance. Finally, we propose Adaptive Layer-wise Distillation (ALD) to mitigate the imbalance in distillation effects varying with the depth of layers. This method assigns greater weight to the losses in layers where the training discrepancies between the teacher and student models are larger during distillation. Through these strategies, [CLS]-KD consistently surpasses existing state-of-the-art methods on the ImageNet-1K dataset across various teacher-student configurations. Furthermore, the proposed method demonstrates its generalization capability through transfer learning experiments on the CIFAR-10, CIFAR-100, and CALTECH-256 datasets.

Communication-Efficient and Privacy-Preserving Federated Learning via Joint Knowledge Distillation and Differential Privacy in Bandwidth-Constrained Networks

Communication-Efficient and Privacy-Preserving Federated Learning via Joint Knowledge Distillation and Differential Privacy in Bandwidth-Constrained Networks

A personalized federated cloud-edge collaboration framework via cross-client knowledge distillation

A personalized federated cloud-edge collaboration framework via cross-client knowledge distillation

Dynamic multi teacher knowledge distillation for semantic parsing in KBQA

Knowledge base question answering (KBQA) is an important task of extracting answers from a knowledge base by analyzing natural language questions. Semantic parsing methods convert natural language questions into executable logical forms to obtain answers on the knowledge base. Conventional approaches often prioritize singular logical forms, overlooking the distinct strengths inherent in various logical frameworks for problem solving. Recognizing that different logical forms may excel in addressing specific types of questions, our aim is to harness these strengths. By integrating the strengths of different logical forms, we expect to achieve more comprehensive and effective semantic parsing solutions. In our paper, we propose a Dynamic Multi Teacher Knowledge Distillation for Semantic Parsing (DMTKD-SP). DMTKD-SP leverages a collection of teacher models, each mastering a unique logical form, to collaboratively guide a student model so that knowledge from different logical forms can be transferred into the student model. To achieve this, we employ a confidence-based weight assignment module to dynamically assign weights for each teacher model. Furthermore, we introduce a self-distillation mechanism to mitigate the confusion caused by simultaneous learning from multiple teachers. We evaluate DMTKD-SP across variants of the KQA Pro dataset, demonstrating an accuracy improvement of 0.35% on five types of questions, with a notable 0.75% improvement for Count questions.