Teacher Network Research Articles

Surface Defect Detection System for Carrot Combine Harvest Based on Multi-Stage Knowledge Distillation

Carrots are a type of vegetable with high nutrition. Before entering the market, the surface defect detection and sorting of carrots can greatly improve food safety and quality. To detect defects on the surfaces of carrots during combine harvest stage, this study proposed an improved knowledge distillation network structure that took yolo-v5s as the teacher network and a lightweight network that replaced the backbone network with mobilenetv2 and completed channel pruning as a student network (mobile-slimv5s). To make the improved student network adapt to the image blur caused by the vibration of the carrot combine harvester, we put the ordinary dataset Dataset (T) and dataset Dataset (S), which contains motion blurring treatment, into the teacher network and the improved lightweight network, respectively, for learning. By connecting multi-stage features of the teacher network, knowledge distillation was carried out, and different weight values were set for each feature to realize that the multi-stage features of the teacher network guide the single-layer output of the student network. Finally, the optimal lightweight network mobile-slimv5s was established, with a network model size of 5.37 MB. The experimental results show that when the learning rate is set to 0.0001, the batch size is set to 64, and the dropout is set to 0.65, the model accuracy of mobile-slimv5s is 90.7%, which is significantly higher than other algorithms. It can synchronously realize carrot harvesting and surface defect detection. This study laid a theoretical foundation for applying knowledge distillation structures to the simultaneous operations of crop combine harvesting and surface defect detection in a field environment. This study effectively improves the accuracy of crop sorting in the field and contributes to the development of smart agriculture.

Semi-XctNet: Volumetric images reconstruction network from a single projection image via semi-supervised learning

Deep learning networks have achieved remarkable progress in various tasks of medical imaging. Most of the recent success in computer vision highly depend on large amounts of carefully annotated data, whereas labelling is arduous, time-consuming and in need of expertise. In this paper, a semi-supervised learning method, Semi-XctNet, is proposed for volumetric images reconstruction from a single X-ray image. In our framework, the effect of regularization on pixel-level prediction is enhanced by introducing a transformation consistent strategy into the model. Furthermore, a multi-stage training strategy is designed to ameliorate the generalization performance of the teacher network. An assistant module is also introduced to improve the pixel quality of pseudo-labels, thereby further improving the reconstruction accuracy of the semi-supervised model. The semi-supervised method proposed in this paper has been extensively validated on the LIDC-IDRI lung cancer detection public data set. Quantitative results show that SSIM (structural similarity measurement) and PSNR (peak signal noise ratio) are 0.8384 and 28.7344 respectively. Compared with the state-of-the-arts, Semi-XctNet exhibits excellent reconstruction performance, thus demonstrating the effectiveness of our method on the task of volumetric images reconstruction network from a single X-ray image.

Egocentric Early Action Prediction via Adversarial Knowledge Distillation

Egocentric early action prediction aims to recognize actions from the first-person view by only observing a partial video segment, which is challenging due to the limited context information of the partial video. In this article, to tackle the egocentric early action prediction problem, we propose a novel multi-modal adversarial knowledge distillation framework. In particular, our approach involves a teacher network to learn the enhanced representation of the partial video by considering the future unobserved video segment, and a student network to mimic the teacher network to produce the powerful representation of the partial video and based on that predicting the action label. To promote the knowledge distillation between the teacher and the student network, we seamlessly integrate adversarial learning with latent and discriminative knowledge regularizations encouraging the learned representations of the partial video to be more informative and discriminative toward the action prediction. Finally, we devise a multi-modal fusion module toward comprehensively predicting the action label. Extensive experiments on two public egocentric datasets validate the superiority of our method over the state-of-the-art methods. We have released the codes and involved parameters to benefit other researchers. 1

Adversarial learning-based multi-level dense-transmission knowledge distillation for AP-ROP detection.

The Aggressive Posterior Retinopathy of Prematurity (AP-ROP) is the major cause of blindness for premature infants. The automatic diagnosis method has become an important tool for detecting AP-ROP. However, most existing automatic diagnosis methods were with heavy complexity, which hinders the development of the detecting devices. Hence, a small network (student network) with a high imitation ability is exactly needed, which can mimic a large network (teacher network) with promising diagnostic performance. Also, if the student network is too small due to the increasing gap between teacher and student networks, the diagnostic performance will drop. To tackle the above issues, we propose a novel adversarial learning-based multi-level dense knowledge distillation method for detecting AP-ROP. Specifically, the pre-trained teacher network is utilized to train multiple intermediate-size networks (i.e., teacher-assistant networks) and one student network by dense transmission mode, where the knowledge from all upper-level networks is transmitted to the current lower-level network. To ensure that two adjacent networks can distill the abundant knowledge, the adversarial learning module is leveraged to enforce the lower-level network to generate the features that are similar to those of the upper-level network. Extensive experiments demonstrate that our proposed method can realize the effective knowledge distillation from the teacher to student networks. We achieve a promising knowledge distillation performance for our private dataset and a public dataset, which can provide a new insight for devising lightweight detecting systems of fundus diseases for practical use.

MDFlow: Unsupervised Optical Flow Learning by Reliable Mutual Knowledge Distillation

Recent works have shown that optical flow can be learned by deep networks from unlabelled image pairs based on brightness constancy assumption and smoothness prior. Current approaches additionally impose an augmentation regularization term for continual self-supervision, which has been proved to be effective on difficult matching regions. However, this method also amplify the inevitable mismatch in unsupervised setting, blocking the learning process towards optimal solution. To break the dilemma, we propose a novel mutual distillation framework to transfer reliable knowledge back and forth between the teacher and student networks for alternate improvement. Concretely, taking estimation of off-the-shelf unsupervised approach as pseudo labels, our insight locates at defining a confidence selection mechanism to extract relative good matches, and then add diverse data augmentation for distilling adequate and reliable knowledge from teacher to student. Thanks to the decouple nature of our method, we can choose a stronger student architecture for sufficient learning. Finally, better student prediction is adopted to transfer knowledge back to the efficient teacher without additional costs in real deployment. Rather than formulating it as a supervised task, we find that introducing an extra unsupervised term for multi-target learning achieves best final results. Extensive experiments show that our approach, termed MDFlow, achieves state-of-the-art real-time accuracy and generalization ability on challenging benchmarks. Code is available at https://github.com/ltkong218/MDFlow.

Strategi komunikasi politik elit NU Pada pilkada lombok tengah 2020 (Studi atas keterlibatan elit NU dalam tim pemenangan paslon Maiq Meres)

The involvement of NU organizational elites can be seen from the role, contribution and political communication strategies they played during the 2020 regional elections in convincing the electorate at the grassroots level. In the regional elections in Central Lombok district in 2020, the political phenomenon that was visible before the voting public was related to the involvement of the Central Lombok NU elite in winning the Lalu Pathul Bahri Candidate Pair - HM Nursiah or the Maiq Meres candidate pair. This research is participatory field research with a descriptive qualitative approach. Data collection was carried out by means of observation, interviews and documentation. The research process was carried out intensively for two months involving informants consisting of main informants and supporting informants regarding the winning of regional head candidates Lalu Pathul Bahri-HM Nursiah. This study found that the Central Lombok district NU elite played an active role as a medium, or media for political communication, the pair of candidates for regent and candidate for deputy regent of Central Lombok Pathul Bhari and HM. Nursiah. The Central Lombok NU elite took part in visiting the electorate, both personally and institutionally. The communication strategy used by the NU elite to convey political messages is to maximize the role of the organizational network, the network of NU teachers, as well as NU's amaliyah activities, which are driven to support pairs of regional head candidates supported by NU.

SD-HRNet: Slimming and Distilling High-Resolution Network for Efficient Face Alignment.

Face alignment is widely used in high-level face analysis applications, such as human activity recognition and human-computer interaction. However, most existing models involve a large number of parameters and are computationally inefficient in practical applications. In this paper, we aim to build a lightweight facial landmark detector by proposing a network-level architecture-slimming method. Concretely, we introduce a selective feature fusion mechanism to quantify and prune redundant transformation and aggregation operations in a high-resolution supernetwork. Moreover, we develop a triple knowledge distillation scheme to further refine a slimmed network, where two peer student networks could learn the implicit landmark distributions from each other while absorbing the knowledge from a teacher network. Extensive experiments on challenging benchmarks, including 300W, COFW, and WFLW, demonstrate that our approach achieves competitive performance with a better trade-off between the number of parameters (0.98 M-1.32 M) and the number of floating-point operations (0.59 G-0.6 G) when compared to recent state-of-the-art methods.

Teacher or supervisor? Effective online knowledge distillation via guided collaborative learning

Knowledge distillation is a widely-used and effective technique to boost the performance of a lightweight student network, by having it mimic the behavior of a more powerful teacher network. This paper presents an end-to-end online knowledge distillation strategy, in which several peer students are trained together and their predictions are aggregated into a powerful teacher ensemble via an effective ensembling technique that uses an online supervisor network to determine the optimal way of combining the student logits. Intuitively, this supervisor network learns the area of expertise of each student and assigns a weight to each student accordingly►it has knowledge of the input image, the ground truth data, and the predictions of each individual student, and tries to answer the following question: “how much can we rely on each student’s prediction, given the current input image with this ground truth class?”. The proposed technique can be thought of as an inference optimization mechanism as it improves the overall accuracy over the same number of parameters. The experiments we performed show that the proposed knowledge distillation consistently improves the performance of the knowledge-distilled students vs. the independently trained students.

Total-body low-dose CT image denoising using a prior knowledge transfer technique with a contrastive regularization mechanism.

Reducing the radiation exposure experienced by patients in total-body computed tomography (CT) imaging has attracted extensive attention in the medical imaging community. A low radiation dose may result in increased noise and artifacts that greatly affect the subsequent clinical diagnosis. To obtain high-quality total-body low-dose CT (LDCT) images, previous deep learning-based research works developed various network architectures. However, most of these methods only employ normal-dose CT (NDCT) images as ground truths to guide the training process of the constructed denoising network. As a result of this simple restriction, the reconstructed images tend to lose favorable image details and easily generate oversmoothed textures. This study explores how to better utilize the information contained in the feature spaces of NDCT images to guide the LDCT image reconstruction process and achieve high-qualityresults. We propose a novel intratask knowledge transfer (KT) method that leverages the knowledge distilled from NDCT images as an auxiliary component of the LDCT image reconstruction process. Our proposed architecture is named the teacher-student consistency network (TSC-Net), which consists of teacher and student networks with identical architectures. By employing the designed KT loss, the student network is encouraged to emulate the teacher network in the representation space and gain robust prior content. In addition, to further exploit the information contained in CT scans, a contrastive regularization mechanism (CRM) built upon contrastive learning is introduced. The CRM aims to minimize and maximize the L2 distances from the predicted CT images to the NDCT samples and to the LDCT samples in the latent space, respectively. Moreover, based on attention and the deformable convolution approach, we design a dynamic enhancement module (DEM) to improve the network capability to transform input informationflows. By conducting ablation studies, we prove the effectiveness of the proposed KT loss, CRM, and DEM. Extensive experimental results demonstrate that the TSC-Net outperforms the state-of-the-art methods in both quantitative and qualitative evaluations. Additionally, the excellent results obtained for clinical readings also prove that our proposed method can reconstruct high-quality CT images for clinicalapplications. Based on the experimental results and clinical readings, the TSC-Net has better performance than other approaches. In our future work, we may explore the reconstruction of LDCT images by fusing the positron emission tomography (PET) and CT modalities to further improve the visual quality of the reconstructed CTimages.

Computation-Efficient Knowledge Distillation via Uncertainty-Aware Mixup

Knowledge distillation (KD) has emerged as an essential technique not only for model compression, but also other learning tasks such as continual learning. Given the richer application spectrum and potential online usage of KD, knowledge distillation efficiency becomes a pivotal component. In this work, we study this little-explored but important topic. Unlike previous works that focus solely on the accuracy of student network, we attempt to achieve a harder goal – to obtain a performance comparable to conventional KD with a lower computation cost during the transfer. To this end, we present UNcertainty-aware mIXup (UNIX), an effective approach that can reduce transfer cost by 20% to 30% and yet maintain comparable or achieve even better student performance than conventional KD. This is made possible via effective uncertainty sampling and a novel adaptive mixup approach that select informative samples dynamically over ample data and compact knowledge in these samples. We show that our approach inherently performs hard sample mining. We demonstrate the applicability of our approach to improve various existing KD approaches by reducing their queries to a teacher network. Extensive experiments are performed on CIFAR100 and ImageNet. Code and model are available at https://github.com/xuguodong03/UNIXKD.

ILIAC: Efficient classification of degraded images using knowledge distillation with cutout data augmentation

Image classification is extensively used in various applications such as satellite imagery, autonomous driving, smartphones, and healthcare. Most of the images used to train classification models can be considered ideal, i.e., without any degradation either due to corruption of pixels in the camera sensors, sudden shake blur, or the compression of images in a specific format. In this paper, we have proposed a novel CNN-based architecture for image classification of degraded images based on intermediate layer knowledge distillation and data augmentation approach cutout named ILIAC. Our approach achieves 1.1%, and 0.4% mean accuracy improvements for all the degradation levels of JPEG and AWGN, respectively, compared to the current state-of-the-art approach. Furthermore, ILIAC method is efficient in computational capacity, i.e., about half the size of the previous state-of-the-art approach in terms of model parameters and GFlops count. Additionally, we demonstrate that we do not necessarily need a larger teacher network in knowledge distillation to improve the model performance and generalization of a smaller student network for the classification of degraded images.

Nurturing “A Specific Kind of Unicorn-y Teacher”: How Teacher Activist Networks Influence the Professional Identity and Practices of Teachers of Color

ABSTRACT This article highlights the ways justice-oriented activist teachers of Color nurture themselves professionally through their involvement in critical professional development through activist teacher networks. This study conducted narrative inquiries of 26 activist teachers of Color across the United States. The counter-stories told by the teachers in this study reflect critical intellectual engagement and highlight specific ways in which these organizations further teacher sustainability, nurturance, and social change. This unified portrait shows a unique interplay between support of moral purpose, intellectual inquiry, and democratic change.

FCKDNet: A Feature Condensation Knowledge Distillation Network for Semantic Segmentation.

As a popular research subject in the field of computer vision, knowledge distillation (KD) is widely used in semantic segmentation (SS). However, based on the learning paradigm of the teacher-student model, the poor quality of teacher network feature knowledge still hinders the development of KD technology. In this paper, we investigate the output features of the teacher-student network and propose a feature condensation-based KD network (FCKDNet), which reduces pseudo-knowledge transfer in the teacher-student network. First, combined with the pixel information entropy calculation rule, we design a feature condensation method to separate the foreground feature knowledge from the background noise of the teacher network outputs. Then, the obtained feature condensation matrix is applied to the original outputs of the teacher and student networks to improve the feature representation capability. In addition, after performing feature condensation on the teacher network, we propose a soft enhancement method of features based on spatial and channel dimensions to improve the dependency of pixels in the feature maps. Finally, we divide the outputs of the teacher network into spatial condensation features and channel condensation features and perform distillation loss calculation with the student network separately to assist the student network to converge faster. Extensive experiments on the public datasets Pascal VOC and Cityscapes demonstrate that our proposed method improves the baseline by 3.16% and 2.98% in terms of mAcc, and 2.03% and 2.30% in terms of mIoU, respectively, and has better segmentation performance and robustness than the mainstream methods.

An Intelligent Question-Answering Model over Educational Knowledge Graph for Sustainable Urban Living

With the development of education informatization and the accumulation of massive educational resources and teaching data in urban environments, educational knowledge graphs that provide good conditions for developing data-driven intelligent education have been proposed. Based on such educational knowledge graphs, the question-answering method can provide students with immediate coaching and significantly increase their learning interest and productivity. However, there is little research on knowledge graph question-answering focused on the educational field. Students tend to consult complex questions that require reasoning; however, the existing QA system cannot satisfy their complex information needs. To help improve sustainable learning efficiency, we propose a novel intelligent question-answering model applied in smart cities, which can reason over the educational knowledge graph to locate the answers to given questions. Our approach uses a highly expressive bilinear graph neural network technology to perform forward reasoning, utilizing the contextual information between graph nodes to improve reasoning ability. On this basis, we propose two-teacher knowledge distillation. We construct two distinct teacher networks by combining forward and backward reasoning, then incorporate the intermediate supervision signals from the two networks to guide the reasoning process, thereby mitigating the phenomenon of spurious path reasoning. Extensive experiments on the MOOC Q&A dataset prove the effectiveness of our approach.

Multi-attention guided and feature enhancement network for vehicle re-identification

Vehicle Re-Identification (Re-ID) aims to discover and match target vehicles in different cameras of road surveillance. The high similarity between vehicle appearances and the dramatic variations in viewpoints and illumination cause great challenges for vehicle Re-ID. Meanwhile, in safety supervision and intelligent traffic systems, one needs a quick efficient method of identifying target vehicles. In this paper, we propose a Multi-Attention Guided Feature Enhancement Network (MAFEN) to extract robust vehicle appearance features. Specifically, the Fusing Spatial-Channel information multi-receptive fields Feature Enhancement module (FSCFE) is first proposed to aggregate richer and more representative multi-receptive fields features at different receptive fields sizes. It also learned the spatial structure information and channel dependencies of the multi-receptive fields features and embedded them to enhance the feature. Then, we construct the Spatial Attention-Guided Adaptive Feature Erasure (SAAFE) module, which uses spatial attention to erase the most distinguishing features. The network’s attention is shifted to potentially salient features to strengthen the ability of the network to extract salient features. In addition, a multi-loss knowledge distillation (MLKD) method using MAFEN as a teacher network is designed to improve computational efficiency. It uses multiple loss functions to jointly supervise the student network. Experimental results on three public datasets demonstrate the merits of the proposed method over the state-of-the-art methods.

Toward Extremely Lightweight Distracted Driver Recognition With Distillation-Based Neural Architecture Search and Knowledge Transfer

The number of traffic accidents has been continuously increasing in recent years worldwide. Many accidents are caused by distracted drivers, who take their attention away from driving. Motivated by the success of Convolutional Neural Networks (CNNs) in computer vision, many researchers developed CNN-based algorithms to recognize distracted driving from a dashcam and warn the driver against unsafe behaviors. However, current models have too many parameters, which is unfeasible for vehicle-mounted computing. This work proposes a novel knowledge-distillation-based framework to solve this problem. The proposed framework first constructs a high-performance teacher network by progressively strengthening the robustness to illumination changes from shallow to deep layers of a CNN. Then, the teacher network is used to guide the architecture searching process of a student network through knowledge distillation. After that, we use the teacher network again to transfer knowledge to the student network by knowledge distillation. Experimental results on the Statefarm Distracted Driver Detection Dataset and AUC Distracted Driver Dataset show that the proposed approach is highly effective for recognizing distracted driving behaviors from photos: (1) the teacher network's accuracy surpasses the previous best accuracy; (2) the student network achieves very high accuracy with only 0.42M parameters (around 55% of the previous most lightweight model). Furthermore, the student network architecture can be extended to a spatial-temporal 3D CNN for recognizing distracted driving from video clips. The 3D student network largely surpasses the previous best accuracy with only 2.03M parameters on the Drive&Act Dataset. The source code is available at https://github.com/Dichao-Liu/Lightweight_Distracted_Driver_Recognition_with_Distillation-Based_NAS_and_Knowledge_Transfer.

Defect Detection Method Based on Knowledge Distillation

Aiming at the problem that traditional surface detection is easily affected by complex industrial environments and cannot extract effective features, a deep learning-based knowledge distillation anomaly detection model is proposed. Firstly, a pre-trained teacher network was used to transfer knowledge of normal samples to the student network in the training phase. In the testing phase, defect detection was achieved based on the feature differences in the output of the teacher-student network for abnormal samples. Secondly, the attention mechanism module and the feature fusion module were added to the teacher network, which enhanced the detection ability of various defects of different sizes and increased the difference in output features between the teacher network and the student network. Finally, the defect image was located at the pixel level. Compared with the advanced knowledge distillation methods, the experimental results of the proposed model on the detection on MVTecAD and the anomaly localization on MVTecAD showed that the method in this paper improved to 90.3% and 90.1% on AUROC respectively, which verified the effectiveness of the method.

Self-Parameter Distillation Dehazing.

In this paper, we propose a novel dehazing method based on self-distillation. In contrast to conventional knowledge distillation approaches that transfer large models (teacher networks) to small models (student networks), we introduce a single knowledge distillation network that transfers network parameters to itself for dehazing. In the early stages, the proposed network transfers scene content (identity) information to the next stage of itself using haze-free data. However, in the later stages, the network transfers haze information to itself using haze data, enabling the accurate dehazing of input images using scene information from the early stages. In a single network, parameters are seamlessly updated from extracting global scene features to dehazing the scene. During the training, forward propagation acts as a teacher network, whereas backward propagation acts as a student network. The experimental results demonstrate that the proposed method considerably outperforms other state-of-the-art dehazing methods.

Binarized Neural Network With Parameterized Weight Clipping and Quantization Gap Minimization for Online Knowledge Distillation

As the applications for artificial intelligence are growing rapidly, numerous network compression algorithms have been developed to restrict computing resources such as smartphones, edge, and IoT devices. Knowledge distillation (KD) leverages soft labels derived from a teacher model to a less parameterized model achieving high accuracy with reduced computational burden. Moreover, online KD provides parallel computing through collaborative learning between teacher and student networks, thus enhancing the training speed. A binarized neural network (BNN) offers an intriguing opportunity to facilitate aggressive compression at the expense of drastically degraded accuracy. In this study, two performance improvements are proposed for online KD when a BNN is applied as a student network: 1) parameterized weight clipping (PWC) to reduce dead weights in the student network and 2) quantization gap-aware adaptive temperature scheduling between the teacher and student networks. In contrast to constant weight clipping (CWC), PWC demonstrates a 3.78% top-1 test accuracy enhancement with trainable weight clipping by decreasing the gradient mismatch with CIFAR-10 dataset. Furthermore, the quantization gap-aware temperature scheduling increases the top-1 test accuracy by 0.08% over online KD at a constant temperature. By aggregating both methodologies, the top-1 test accuracy for CIFAR-10 dataset was 94.60%, and that for Tiny-ImageNet dataset was comparable to that of the 32-bit full-precision neural network.

Deep Generative Knowledge Distillation by Likelihood Finetuning

Knowledge Distillation (KD) is designed to train smaller student models using a larger pretrained teacher model. However, in decentralized data systems such as blockchain, privacy concerns may arise, making the data inaccessible. To address this issue, Data-Free KD (DFKD) methods have been proposed, which extract prior knowledge from teacher networks and use it to synthesize data for KD. Previous DFKD methods faced challenges due to the large search space of data generation. Recently, deep generative models (DGMs) have been proposed to learn data distribution using deep networks, which provides an efficient way to reduce the search space by generating a set of pseudo data. In this paper, we explore the performance of KD trained using pseudo samples generated by pretrained DGMs and find that the correlation with image quality is not always positive. Based on this observation, we propose a new DFKD framework called Generative Knowledge Distillation (GenKD) that reduces the search space by constructing a prior distribution modeled by DGMs for their power of likelihood estimation. Specifically, we use energy-based models (EBM) to generate data from the Maximum Likelihood Estimation (MLE) of the EBM and gradients from downstream KD tasks by policy gradient. We then train the student model using the pretrained teacher model and pseudo samples. We also implement our GenKD framework on several widely-used benchmarks, including CIFAR100, CIFAR10, and SVHN. Our experiments demonstrate that we can generate high-quality pseudo samples quantitatively and qualitatively using GenKD. Additionally, the top-1 accuracy of the student network can approach state-of-the-art (SOTA) DFKD methods trained using fewer pseudo samples and less generation time.