Distillation Framework Research Articles

Self-knowledge distillation for surgical phase recognition.

Advances in surgical phase recognition are generally led by training deeper networks. Rather than going further with a more complex solution, we believe that current models can be exploited better. We propose a self-knowledge distillation framework that can be integrated into current state-of-the-art (SOTA) models without requiring any extra complexity to the models or annotations. Knowledge distillation is a framework for network regularization where knowledge is distilled from a teacher network to a student network. In self-knowledge distillation, the student model becomes the teacher such that the network learns from itself. Most phase recognition models follow an encoder-decoder framework. Our framework utilizes self-knowledge distillation in both stages. The teacher model guides the training process of the student model to extract enhanced feature representations from the encoder and build a more robust temporal decoder to tackle the over-segmentation problem. We validate our proposed framework on the public dataset Cholec80. Our framework is embedded on top of four popular SOTA approaches and consistently improves their performance. Specifically, our best GRU model boosts performance by [Formula: see text] accuracy and [Formula: see text] F1-score over the same baseline model. We embed a self-knowledge distillation framework for the first time in the surgical phase recognition training pipeline. Experimental results demonstrate that our simple yet powerful framework can improve performance of existing phase recognition models. Moreover, our extensive experiments show that even with 75% of the training set we still achieve performance on par with the same baseline model trained on the full set.

A method of knowledge distillation based on feature fusion and attention mechanism for complex traffic scenes

Object detectors based on deep learning can run smoothly on a terminal device in complex traffic scenes, and the model compression method has become a research hotspot. Considering student network single learning in the knowledge distillation algorithm, the dependence on loss function design leads to parameter sensitivity and other problems, we propose a new knowledge distillation method with second-order term attention mechanisms and feature fusion of adjacent layers. First, we build a knowledge distillation framework based on YOLOv5 and propose a new attention mechanism in the teacher network backbone to extract the hot map. Then, we combine the hot map features with the next level features through the fusion module. By fusing the useful information of the low convolution layer and the feature map of the high convolution layer to help the student network obtain the final prediction map. Finally, to improve the accuracy of small objects, we add a 160 × 160 detection head and use a transformer encoder block module to replace the convolution network of the head. Sufficient experimental results show that our method achieves state-of-the-art performance. The speed and number of parameters remain unchanged, but the average detection accuracy is 97.4% on the KITTI test set. On the Cityscapes test set, the average detection accuracy reaches 92.7%.

A knowledge distillation-based deep interaction compressed network for CTR prediction

Click-through rate (CTR) prediction is an essential task in online advertising recommender systems. As a hot research frontier driven by academic and industrial needs, CTR prediction has observed remarkable progress in recent years. How to effectively learn sophisticated feature interactions hidden behind user behaviors is critical in maximizing CTR. Targeted at the above issue, this paper first constructs a deep interaction compressed network model (abbreviated as DICN) that aims to generate both low- and high-order feature interactions with different weights in both explicit and implicit fashions. Then, a KD-DICN model is created using a knowledge distillation framework to reduce the complexity of DICN. Finally, comprehensive experiments are conducted on two public datasets to demonstrate the effectiveness and efficiency of our models, which outperform the existing state-of-the-art methods. Therefore, the research presented in this paper can not only enrich and develop the theories related to CTR prediction, but it can also assist some advertising industries in better defining their service audience and recommending more popular contents to consumers more accurately.

INFER: Distilling knowledge from human-generated rules with uncertainty for STINs

As a long-time wish, researchers always want to find a way to fuse human knowledge directly into machine models that can fulfill intelligent tasks. Existing researches attempted to approach this goal by manually labeling data or writing simple rules without considering the human's inherent ability to estimate uncertainty. These approaches cannot take full advantage of human abilities and hence make the overall system inefficient. In this paper, we propose a novel approach INFER that can distill knowledge from humans in the form of rules with uncertainty. Firstly, we propose a new paradigm of providing human intelligence by expressing human knowledge as uncertain rules. After obtaining the set of rules, we design a rule aggregation model and a classification model, these two models are jointly trained in a knowledge distillation framework with the uncertainty knowledge. We further improve the distillation process with a curriculum learning-based training method. Through these, we can directly extract knowledge from inaccurate human knowledge. Empirical results on four different tasks demonstrate our proposed INFER approach can significantly improve model performance. Furthermore, the proposed uncertainty can provide more information and be effective in refining rules.

Dual cross knowledge distillation for image super-resolution

The huge computational requirements and memory footprint limit the practical deployment of super resolution (SR) models. Knowledge distillation (KD) allows student networks to obtain performance improvement by learning from over-parameterized teacher networks. Previous work has attempted to solve SR distillation problem by using feature-based distillation, which ignores the supervisory role of the teacher module itself. In this paper, we introduce a cross knowledge distillation framework to compress and accelerate SR models. Specifically, we propose to obtain supervision by cascading the student into the teacher network for directly utilizing teacher’s well-trained parameters. This not only reduces the difficulty of optimization for students but also avoids designing alignment with obscure feature textures between two networks. To the best of our knowledge, we are the first work to explore the cross distillation paradigm on the SR tasks. Experiments on typical SR networks have shown the superiority of our method in generated images, PSNR and SSIM.

Customizing a teacher for feature distillation

Knowledge distillation is a method to train a lightweight network by transferring class probability knowledge from a cumbersome teacher network. However, transferring only the class probability knowledge would limit the distillation performance. Therefore, several approaches have been proposed to transfer the teacher's knowledge at the feature map level. In this paper, we revisit the feature distillation method and have found that the larger the teacher's architecture/capacity becomes, the more difficult it is for the student to imitate. Thus, the feature distillation method is unable to achieve its full potential. To address this, a novel end-to-end distillation framework, termed Customizing a Teacher for Feature Distillation (CTFD), is proposed to train a teacher to be more compatible with its student. In addition, we apply the customized teacher to three feature distillation methods. Moreover, data augmentation is used as a trick to train the student to improve its generalization performance. Extensive empirical experiments and analyses are conducted on three computer vision tasks, including image classification, transfer learning, and object detection, to substantiate the effectiveness of the proposed method.

A Cooperative Vehicle-Infrastructure System for Road Hazards Detection With Edge Intelligence

Road hazards (RH) have always been the cause of many serious traffic accidents. These have posed a threat to the safety of drivers, passengers, and pedestrians, and have also resulted in significant losses to people and even to the economies of countries. Hence, road hazards detection (RHD) could play an essential role in intelligent transportation systems (ITSITS). The cooperative vehicle-infrastructure systems (CVIS) coordinate the communication between vehicles and roadside infrastructures. Onboard computing devices (OCD), then, make fast analyses and decisions based on road conditions. In this study, an RHD solution based on CVIS is proposed. Firstly, a high-performance heavy action detection model is selected. Using a meta-learning paradigm, critical features are generalized from a few-shot RH data. Secondly, we designed a lightweight RHD model to ensure its smooth inference on an OCD. Thirdly, we use a knowledge distillation (KD) framework to progressively distill the features of the complex model and the privileged information of the data into the lightweight one. Experimental results demonstrate that the model can effectively detect RH and obtain an accuracy of 90.2% with an inference time of 14.7ms.

Multi-target Knowledge Distillation via Student Self-reflection

Knowledge distillation is a simple yet effective technique for deep model compression, which aims to transfer the knowledge learned by a large teacher model to a small student model. To mimic how the teacher teaches the student, existing knowledge distillation methods mainly adapt an unidirectional knowledge transfer, where the knowledge extracted from different intermedicate layers of the teacher model is used to guide the student model. However, it turns out that the students can learn more effectively through multi-stage learning with a self-reflection in the real-world education scenario, which is nevertheless ignored by current knowledge distillation methods. Inspired by this, we devise a new knowledge distillation framework entitled multi-target knowledge distillation via student self-reflection or MTKD-SSR, which can not only enhance the teacher’s ability in unfolding the knowledge to be distilled, but also improve the student’s capacity of digesting the knowledge. Specifically, the proposed framework consists of three target knowledge distillation mechanisms: a stage-wise channel distillation (SCD), a stage-wise response distillation (SRD), and a cross-stage review distillation (CRD), where SCD and SRD transfer feature-based knowledge (i.e., channel features) and response-based knowledge (i.e., logits) at different stages, respectively; and CRD encourages the student model to conduct self-reflective learning after each stage by a self-distillation of the response-based knowledge. Experimental results on five popular visual recognition datasets, CIFAR-100, Market-1501, CUB200-2011, ImageNet, and Pascal VOC, demonstrate that the proposed framework significantly outperforms recent state-of-the-art knowledge distillation methods.

Hybrid KD-NFT: A multi-layered NFT assisted robust Knowledge Distillation framework for Internet of Things

The Internet of Things (IoT) concept has increasingly gained popularity among different industrial and scientific fields by leveraging the development of modern electronic chip manufacturing and networking hardware and techniques. As the combination of end devices and networking techniques, it becomes more and more possible that people can organize an IoT system to serve more extensive industrial and scientific needs by running a higher level of machine learning models, such as Federated Learning (FL) and Knowledge Distillation (KD) architectures. However, security over network communications is vital when the whole system spreads to massive scales in networking, including data integrity and robustness over external malicious attacks, which would significantly affect the system’s overall modal training effectiveness. In this paper, we propose an ungraded version of the Non-Fungible Token (NFT) assisted Knowledge Distillation framework, aiming at leveraging the blockchain features on data security to solve the intrinsic robustness defects in a naive KD architecture and reducing overall processing time by adding a local blockchain layer. Our major contributions can be concluded as the following (1) a revised version Hybrid KD-NFT framework from our previously proposed State-of-the-art (SOTA) framework, which improves the overall effectiveness and reduces the overall latency when compared to its original version; (2) the proposal and implementation of a hybrid (combination of private and public chain) multi-layer blockchain architecture; and (3) conducting more sets of experiments and performing more comprehensive comparisons over existing other KD frameworks in more metrics to prove our proposed Hybrid KD-NFT framework is a valid work.

A 3DCNN-Based Knowledge Distillation Framework for Human Activity Recognition.

Human action recognition has been actively explored over the past two decades to further advancements in video analytics domain. Numerous research studies have been conducted to investigate the complex sequential patterns of human actions in video streams. In this paper, we propose a knowledge distillation framework, which distills spatio-temporal knowledge from a large teacher model to a lightweight student model using an offline knowledge distillation technique. The proposed offline knowledge distillation framework takes two models: a large pre-trained 3DCNN (three-dimensional convolutional neural network) teacher model and a lightweight 3DCNN student model (i.e., the teacher model is pre-trained on the same dataset on which the student model is to be trained on). During offline knowledge distillation training, the distillation algorithm trains only the student model to help enable the student model to achieve the same level of prediction accuracy as the teacher model. To evaluate the performance of the proposed method, we conduct extensive experiments on four benchmark human action datasets. The obtained quantitative results verify the efficiency and robustness of the proposed method over the state-of-the-art human action recognition methods by obtaining up to 35% improvement in accuracy over existing methods. Furthermore, we evaluate the inference time of the proposed method and compare the obtained results with the inference time of the state-of-the-art methods. Experimental results reveal that the proposed method attains an improvement of up to 50× in terms of frames per seconds (FPS) over the state-of-the-art methods. The short inference time and high accuracy make our proposed framework suitable for human activity recognition in real-time applications.

Graph Flow: Cross-layer Graph Flow Distillation for Dual Efficient Medical Image Segmentation.

With the development of deep convolutional neural networks, medical image segmentation has achieved a series of breakthroughs in recent years. However, high-performance convolutional neural networks always mean numerous parameters and high computation costs, which will hinder the applications in resource-limited medical scenarios. Meanwhile, the scarceness of large-scale annotated medical image datasets further impedes the application of high-performance networks. To tackle these problems, we propose Graph Flow, a comprehensive knowledge distillation framework, for both network efficiency and annotation-efficiency medical image segmentation. Specifically, the Graph Flow Distillation transfers the essence of cross-layer variations from a well trained cumbersome teacher network to a non-trained compact student network. In addition, an unsupervised Paraphraser Module is integrated to purify the knowledge of the teacher, which is also beneficial for the training stabilization. Furthermore, we build a unified distillation framework by integrating the adversarial distillation and the vanilla logits distillation, which can further refine the final predictions of the compact network. With different teacher networks (traditional convolutional architecture or prevalent transformer architecture) and student networks, we conduct extensive experiments on four medical image datasets with different modalities (Gastric Cancer, Synapse, BUSI, and CVC-ClinicDB).We demonstrate the prominent ability of our method on these datasets, which achieves competitive performances. Moreover, we demonstrate the effectiveness of our Graph Flow through a novel semi-supervised paradigm for dual efficient medical image segmentation. Our code will be available at Graph Flow.

Lightweight Device-Free Wireless Sensing Using Information-for-Complexity Strategy

Device-free wireless sensing (DFWS) has drawn lots of attention due to its potential application in the fields of human–computer interaction and smart home. Deep networks-based DFWS technique has achieved excellent sensing performance. However, network complexity limits its deployment on resource-limited sensing devices. A feasible way is to implement a simple network to accomplish the DFWS task. However, the sensing performance will drop dramatically due to its limited learning ability. In this article, to realize lightweight DFWS with acceptable performance, we propose an information-for-complexity strategy to promote the learning ability of the simple network. We leverage knowledge distillation framework to explore external information to augment the extrinsic learning ability, and utilize multiscale receptive fields to explore the internal information to augment the intrinsic learning ability. Extensive experiments on a 77 GHz mmWave testbed show that the performance degradation of the lightweight DFWS system is within 3%, while the complexity decreases remarkably.

CEKD: Cross-Modal Edge-Privileged Knowledge Distillation for Semantic Scene Understanding Using Only Thermal Images

Semantic scene understanding using thermal images has received great attention due to the advantage that thermal imaging cameras could see in challenging illumination conditions. However, thermal images are lack of color information and the edges in thermal images are often blurred, making them not very suitable to be directly used by existing semantic segmentation networks that are designed with RGB images. To address this problem, we propose a cross-modal edge-privileged knowledge distillation framework, which utilizes a well-trained RGB-Thermal fusion-based semantic segmentation network with edge-privileged information as the teacher, to guide the training of a semantic segmentation network as the student. The student network only uses thermal images. The experimental results on a public dataset demonstrate that under the guidance of the teacher, the student network achieves superior performance over the state of the arts using only thermal images. Our code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/lab-sun/CEKD</uri> .

Domain-Invariant Feature Progressive Distillation with Adversarial Adaptive Augmentation for Low-Resource Cross-Domain NER

Considering the expensive annotation in Named Entity Recognition (NER ), Cross-domain NER enables NER in low-resource target domains with few or without labeled data, by transferring the knowledge of high-resource domains. However, the discrepancy between different domains causes the domain shift problem and hampers the performance of cross-domain NER in low-resource scenarios. In this article, we first propose an adversarial adaptive augmentation, where we integrate the adversarial strategy into a multi-task learner to augment and qualify domain adaptive data. We extract domain-invariant features of the adaptive data to bridge the cross-domain gap and alleviate the label-sparsity problem simultaneously. Therefore, another important component in this article is the progressive domain-invariant feature distillation framework. A multi-grained MMD (Maximum Mean Discrepancy) approach in the framework to extract the multi-level domain invariant features and enable knowledge transfer across domains through the adversarial adaptive data. Advanced Knowledge Distillation (KD) schema processes progressively domain adaptation through the powerful pre-trained language models and multi-level domain invariant features. Extensive comparative experiments over four English and two Chinese benchmarks show the importance of adversarial augmentation and effective adaptation from high-resource domains to low-resource target domains. Comparison with two vanilla and four latest baselines indicates the state-of-the-art performance and superiority confronted with both zero-resource and minimal-resource scenarios.

Generalized Knowledge Distillation for Unimodal Glioma Segmentation from Multimodal Models

Gliomas, primary brain tumors arising from glial cells, can be effectively identified using Magnetic Resonance Imaging (MRI), a widely employed diagnostic tool in clinical settings. Accurate glioma segmentation, which is crucial for diagnosis and surgical intervention, can be achieved by integrating multiple MRI modalities that offer complementary information. However, limited access to multiple modalities in certain clinical contexts often results in suboptimal performance of glioma segmentation methods. This study introduces a novel generalized knowledge distillation framework designed to transfer multimodal knowledge from a teacher model to a unimodal student model via two distinct distillation strategies: segmentation graph distillation and cascade region attention distillation. The former enables the student to replicate the teacher’s softened output, whereas the latter facilitates extraction and learning of region feature information at various levels within the teacher model. Our evaluation of the proposed distillation strategies using the BraTS 2018 dataset confirms their superior performance in unimodal segmentation contexts compared with existing methods.

Integrated Knowledge Distillation for Efficient One-stage Object Detection Network

Due to the low latency requirements in object detection, numbers of one-stage methods like YOLO and SSD adopt a shared head for both classification and localisation tasks. While the decoupled head used to decouple the subtasks into different heads are getting more popular in one-stage detection because they improve accuracy. In contrast, the computational complexity caused by the decoupled head cant be ignored. To solve these problems, we propose an integrated knowledge distillation framework for transferring the representation ability of the decoupled head to the original coupled head and contributing to efficient one-stage object detection. It solves the problem that the coupled head is insufficient in handling the conflict of subtasks and avoids the time delay introduced by the coupling head and the increase of network parameters.

Ambiguity-aware robust teacher (ART): Enhanced self-knowledge distillation framework with pruned teacher network

Self-knowledge distillation (self-KD) methods, which use a student model itself as the teacher model instead of a large and complex teacher model, are currently a subject of active study. Since most previous self-KD approaches relied on the knowledge of a single teacher model, if the teacher model incorrectly predicted confusing samples, poor-quality knowledge was transferred to the student model. Unfortunately, natural images are often ambiguous for teacher models due to multiple objects, mislabeling, or low quality. In this paper, we propose a novel knowledge distillation framework named ambiguity-aware robust teacher knowledge distillation (ART-KD) that provides refined knowledge, that reflects the ambiguity of the samples with network pruning. Since the pruned teacher model is simply obtained by copying and pruning the teacher model, re-training process is unnecessary in ART-KD. The key insight of ART-KD lies in the predictions of a teacher model and pruned teacher model for ambiguous samples providing different distributions with low similarity. From these two distributions, we obtain a joint distribution considering the ambiguity of the samples as teacher’s knowledge for distillation. We comprehensively evaluate our method on public classification benchmarks, as well as more challenging benchmarks for fine-grained visual recognition (FGVR), achieving much superior performance to state-of-the-art counterparts.

Reverse Knowledge Distillation with Two Teachers for Industrial Defect Detection

Industrial defect detection plays an important role in smart manufacturing and is widely used in various scenarios such as smart inspection and product quality control. Currently, although utilizing a framework for knowledge distillation to identify industrial defects has achieved great progress, it is still a significant challenge task to extract better image features and prevent overfitting for student networks. In this study, a reverse knowledge distillation framework with two teachers is designed. First, for the teacher network, two teachers with different architectures are used to extract the diverse features of the images from multiple models. Second, considering the different contributions of channels and different teacher networks, the attention mechanism and iterative attention feature fusion idea are introduced. Finally, to prevent overfitting, the student network is designed with a network architecture that is inconsistent with the teacher network. Extensive experiments were conducted on Mvtec and BTAD datasets, which are industrial defect detection datasets. On the Mvtec dataset, the average accuracy values of image-level and pixel-level ROC achieved 99.43% and 97.87%, respectively. On the BTAD dataset, the average accuracy values of image-level and pixel-level ROC reached 94% and 98%, respectively. The performance on both datasets is significantly improved, demonstrating the effectiveness of our method.

SSDA-YOLO: Semi-supervised domain adaptive YOLO for cross-domain object detection

Domain adaptive object detection (DAOD) aims to alleviate transfer performance degradation caused by the cross-domain discrepancy. However, most existing DAOD methods are dominated by outdated and computationally intensive two-stage Faster R-CNN, which is not the first choice for industrial applications. In this paper, we propose a novel semi-supervised domain adaptive YOLO (SSDA-YOLO) based method to improve cross-domain detection performance by integrating the compact one-stage stronger detector YOLOv5 with domain adaptation. Specifically, we adapt the knowledge distillation framework with the Mean Teacher model to assist the student model in obtaining instance-level features of the unlabeled target domain. We also utilize the scene style transfer to cross-generate pseudo images in different domains for remedying image-level differences. In addition, an intuitive consistency loss is proposed to further align cross-domain predictions. We evaluate SSDA-YOLO on public benchmarks including PascalVOC, Clipart1k, Cityscapes, and Foggy Cityscapes. Moreover, to verify its generalization, we conduct experiments on yawning detection datasets collected from various real classrooms. The results show considerable improvements of our method in these DAOD tasks, which reveals both the effectiveness of proposed adaptive modules and the urgency of applying more advanced detectors in DAOD. Our code is available on https://github.com/hnuzhy/SSDA-YOLO.

Knowledge distillation-enhanced multitask framework for recommendation

With the ever-growing amount of online information, recommender systems (RSs) act as information filtering tools and are widely used in various e-commerce platforms. Recommender methods generally adopt only one type of behavior data for single-task learning. Multitask learning is commonly used to simultaneously address multiple tasks utilizing as much information as possible. We propose a knowledge distillation-enhanced shared-bottom model for recommender multitask learning. The postview click-through rate, postview conversion rate and average transaction value are learned together to determine the final gross merchandise volume. First, a mixture of experts with gate networks is used as a shared bottom to learn task-specific representations for all tasks, while a tree distillation framework is designed as an expert for better feature selection. Then, we design a neural factorization machine as a task-specific prediction network to estimate individual goals. Finally, an objective function is designed based on task outputs to optimize the gross merchandise volume. Our proposed framework can enhance recommender preferences while avoiding generating a large number of parameters.This model serves the e-commerce scenario, with the following specific use scenario: building personalized GMV predictions for small scenarios on e-commerce platforms at low cost, which is a 0 to 1 process that is often highly rewarding. Ultimately, we propose to improve the reliability of e-commerce scenarios.