Teacher Network Research Articles

Network lightweight method based on knowledge distillation is applied to RV reducer fault diagnosis

As a core component of industrial robots, the RV reducer directly affects the normal operation of the robot, so it is of great significance to monitor its status and diagnose faults. In the field of fault diagnosis, intelligent diagnosis methods based on deep learning have shown great advantages in accuracy and efficiency. However, as the network depth and scale increase, the exponentially growing model computation and parameter amounts require higher hardware requirements for computers, making it difficult to deploy on embedded platforms with limited computing resources. This makes it difficult for deep learning-based fault diagnosis methods to be applied in practical industrial settings that emphasize real-time performance, portability, and accuracy. This paper proposes a network lightweight method based on knowledge distillation (KD). Using the two-dimensional time–frequency map of vibration signals as the model input, the improved MobileNet-V3 network is used as the teacher network, and the simplified convolutional neural network is used as the student network (SN). The method of KD is applied to condense the knowledge and transfer it to the SN. The proposed method is validated using an RV reducer fault simulation experiment platform, and the results show that the proposed method reduces computation and parameter amounts by about 170 times at an accuracy rate of 94.37%, and run time is shortened by nearly one-third, and a generalization verification was conducted using the rotating mechanical fault simulation experiment platform. The models were also deployed on embedded devices to verify that the method proposed in this paper effectively reduces the deep learning network model’s demand for hardware resources of the operating environment. This provides an effective reference for deploying and implementing deep learning-based fault diagnosis on embedded systems with lower hardware configurations.

A method of knowledge distillation based on feature fusion and attention mechanism for complex traffic scenes

Object detectors based on deep learning can run smoothly on a terminal device in complex traffic scenes, and the model compression method has become a research hotspot. Considering student network single learning in the knowledge distillation algorithm, the dependence on loss function design leads to parameter sensitivity and other problems, we propose a new knowledge distillation method with second-order term attention mechanisms and feature fusion of adjacent layers. First, we build a knowledge distillation framework based on YOLOv5 and propose a new attention mechanism in the teacher network backbone to extract the hot map. Then, we combine the hot map features with the next level features through the fusion module. By fusing the useful information of the low convolution layer and the feature map of the high convolution layer to help the student network obtain the final prediction map. Finally, to improve the accuracy of small objects, we add a 160 × 160 detection head and use a transformer encoder block module to replace the convolution network of the head. Sufficient experimental results show that our method achieves state-of-the-art performance. The speed and number of parameters remain unchanged, but the average detection accuracy is 97.4% on the KITTI test set. On the Cityscapes test set, the average detection accuracy reaches 92.7%.

A single stage knowledge distillation network for brain tumor segmentation on limited MR image modalities

Background and objectivePrecisely segmenting brain tumors using multimodal Magnetic Resonance Imaging (MRI) is an essential task for early diagnosis, disease monitoring, and surgical planning. Unfortunately, the complete four image modalities utilized in the well-known BraTS benchmark dataset: T1, T2, Fluid-Attenuated Inversion Recovery (FLAIR), and T1 Contrast-Enhanced (T1CE) are not regularly acquired in clinical practice due to the high cost and long acquisition time. Rather, it is common to utilize limited image modalities for brain tumor segmentation. MethodsIn this paper, we propose a single stage learning of knowledge distillation algorithm that derives information from the missing modalities for better segmentation of brain tumors. Unlike the previous works that adopted a two-stage framework to distill the knowledge from a pre-trained network into a student network, where the latter network is trained on limited image modality, we train both models simultaneously using a single-stage knowledge distillation algorithm. We transfer the information by reducing the redundancy from a teacher network trained on full image modalities to the student network using Barlow Twins loss on a latent-space level. To distill the knowledge on the pixel level, we further employ a deep supervision idea that trains the backbone networks of both teacher and student paths using Cross-Entropy loss. ResultsWe demonstrate that the proposed single-stage knowledge distillation approach enables improving the performance of the student network in each tumor category with overall dice scores of 91.11% for Tumor Core, 89.70% for Enhancing Tumor, and 92.20% for Whole Tumor in the case of only using the FLAIR and T1CE images, outperforming the state-of-the-art segmentation methods. ConclusionsThe outcomes of this work prove the feasibility of exploiting the knowledge distillation in segmenting brain tumors using limited image modalities and hence make it closer to clinical practices.

A novel method for Jinnan cattle individual classification based on deep mutual learning

As the core technology of precision animal husbandry, efficient and rapid identification of Jinnan cattle individuals can promote the scale, informatization and refinement of breeding, which is very necessary for the development of animal husbandry at this stage. However, the traditional livestock individual recognition method based on ear tag is labour-consuming, time-consuming, inefficient, easy to wear and limited by the recognition distance, and the accuracy is also very low. In order to solve this problem, a new method of Jinnan cattle individual recognition based on deep mutual learning is proposed by using the non-contact image recognition method. Two student networks are designed. They supervise each other and complete the task together. Their efficiency can be higher than that of a strong teacher network. Through this method, the individual recognition performance of Jinnan cattle is also enhanced. The experimental results verify the effectiveness of the method of deep mutual learning. Consult and learn from the peer network are used to improve generalization, so as to improve model recognition performance. Finally, the accuracy is 99.3% on the Jinnan cattle individual dataset established in this paper. The application of the contactless cattle individual recognition method in the farm is of great significance.

Mutually aided uncertainty incorporated dual consistency regularization with pseudo label for semi-supervised medical image segmentation

Semi-supervised learning has contributed plenty to promoting computer vision tasks. Especially concerning medical images, semi-supervised image segmentation can significantly reduce the labor and time cost of labeling images. Among the existing semi-supervised methods, pseudo-labelling and consistency regularization prevail; however, the current related methods still need to achieve satisfactory results due to the poor quality of the pseudo-labels generated and needing more certainty awareness the models. To address this problem, we propose a novel method that combines pseudo-labelling with dual consistency regularization based on a high capability of uncertainty awareness. This method leverages a cycle-loss regularized to lead to a more accurate uncertainty estimate. Followed by the uncertainty estimation, the certain region with its pseudo-label is further trained in a supervised manner. In contrast, the uncertain region is used to promote the dual consistency between the student and teacher networks. The developed approach was tested on three public datasets and showed that: 1) The proposed method achieves excellent performance improvement by leveraging unlabeled data; 2) Compared with several state-of-the-art (SOTA) semi-supervised segmentation methods, ours achieved better or comparable performance.

Ability-aware knowledge distillation for resource-constrained embedded devices

Deep Neural Network (DNN) models have notably improved the efficiency of machine learning tasks. However, their high storage and computational costs restrict their deployment on resource-limited embedded devices. Knowledge distillation (KD) has emerged as a promising approach for compressing DNN models. However, two challenges in KD, namely the capacity gap problem and the time-consuming redundancy problem, have hindered its performance and efficiency in compression. To alleviate these challenges, this paper proposes a novel framework, called Ability-Aware Knowledge Distillation (AAKD). AAKD introduces a knowledge sample selection strategy and an adaptive teacher switching strategy based on the dynamic awareness of the student’s ability. This enables the framework to automatically select suitable knowledge samples and teacher networks according to the increasing representation ability of students. Extensive experiments on different datasets and models have demonstrated that AAKD can enhance the performance of compact student models, significantly improve the efficiency of distillation, and lead to higher compression rates.

Super-Resolution of Images using Contrastive Self-Learning

Abstract: Super-resolution (SR) techniques aim to enhance the resolution and quality of low-resolution images. In recent years, deep learning-based approaches have achieved remarkable success in this field. However, training deep SR models often requires large-scale datasets and computationally expensive operations, limiting their practicality. Different approaches have been proposed to tackle computation and time constraints. Our novel approach involves a combination of knowledge distillation and contrastive loss function to train a compact and efficient contrastive self-distillation (CSD) model. In the framework, a teacher network is initially trained on a large dataset using a traditional supervised learning approach. The teacher network learns to generate high-resolution images from low-resolution inputs. Subsequently, a student network is initialized with the same architecture as the teacher network but with fewer parameters. The student network is then trained to mimic the behavior of the teacher network by utilizing a contrastive loss. The contrastive loss is formulated by constructing positive and negative pairs of low-resolution and high-resolution image patches. The proposed CSD method achieves competitive performance compared to state-of-the-art SR methods while requiring significantly fewer parameters and computational resources. Furthermore, it demonstrates improved generalization capability by effectively reconstructing details in real-world images.

Dual cross knowledge distillation for image super-resolution

The huge computational requirements and memory footprint limit the practical deployment of super resolution (SR) models. Knowledge distillation (KD) allows student networks to obtain performance improvement by learning from over-parameterized teacher networks. Previous work has attempted to solve SR distillation problem by using feature-based distillation, which ignores the supervisory role of the teacher module itself. In this paper, we introduce a cross knowledge distillation framework to compress and accelerate SR models. Specifically, we propose to obtain supervision by cascading the student into the teacher network for directly utilizing teacher’s well-trained parameters. This not only reduces the difficulty of optimization for students but also avoids designing alignment with obscure feature textures between two networks. To the best of our knowledge, we are the first work to explore the cross distillation paradigm on the SR tasks. Experiments on typical SR networks have shown the superiority of our method in generated images, PSNR and SSIM.

Semi-supervised deep neural networks for left atrial segmentation from cardiac MRI

Abstract Funding Acknowledgements Type of funding sources: None. Introduction Left atrium size and function is an important but less studied prognostic marker for many cardiovascular diseases [1]. Cardiac Magnetic Resonance Imaging (MRI) can capture atrial structure with a high temporal and spatial resolution. However, diagnoses and accurate measurement of volumes from MRI often rely on time-consuming manual processing, or needs correction of software segmentation error by clinicians. Robust, automated methods are highly desirable to assist morphological analysis of dynamic changes in the heart. Ultimately this facilitates the wider application of cardiac MRI in early identification and prevention of potential cardiovascular disease. Purpose To introduce a semi-supervised teacher-student deep neural networks (requiring less labelled data) for accurate segmentation of left atrium from cardiac MRI images, and compare their performance with conventional supervised learning. Methods The basic structure for a mean teacher deep neural network is shown in Fig.1. Network training starts in a supervised manner. At each MRI batch input, both the student and the teacher model evaluate the images with applied noise. The student network is updated by gradient descent and segmentation loss is obtained by comparing its prediction with the label. The exponential moving average (EMA) of the student weight is then migrated to update the teacher network. Consistency loss between the student’s and teacher’s prediction (filtered by uncertainty map in Uncertainty-Aware Mean Teacher, UA-MT [2]) is obtained to jointly update the student. Unlabeled images continually improve the network without using supervised loss. We compared three semi-supervised teacher-student networks UA-MT, Hierarchical Consistency Regularized Mean Teacher (HCR-MT) and Semi-supervised Contrastive Consistency model (SCC), with the supervised network TransUNnet [3], using benchmark cardiac MRI dataset from MICCAI 2018 Atrial Segmentation Challenge [4]. Out of 20 MRI scans (1760 images), 16 scans were used for training, and 4 scans for testing. For semi-supervised learning, 75% of the training set was used as labelled data and 25% was assumed to be unlabeled. In the supervised learning, all labels were enabled. Results Example segmentation results from three semi-supervised models are shown in Fig 2. For comparison, the fully supervised TransUNnet produced an accuracy of 76%. Benefiting from effective teacher-student learning, the semi-supervised models achieved a superior accuracy of 89%, while also requiring less labelled data. Conclusion The proposed teacher-student deep neural network facilitates effective semi-supervised learning, and outperforms supervised TransUNet in segmenting the left atrium in cardiac MRI. Our semi-supervised approach uses less training data, reducing tedious and time-consuming manual labelling tasks, ultimately unlocking the ability to effectively utilise the full power of deep learning in medical applications.

Distilling EEG representations via capsules for affective computing

Affective computing with Electroencephalogram (EEG) is a challenging task that requires cumbersome models to effectively learn the information contained in large-scale EEG signals, causing difficulties for real-time smart-device deployment. In this paper, we propose a novel knowledge distillation pipeline to distill EEG representations via capsule-based architectures for both classification and regression tasks. Our goal is to distill information from a heavy model to a lightweight model for subject-specific tasks. To this end, we first pre-train a large model (teacher network) on large number of training samples. Then, we employ the teacher network to learn the discriminative features embedded in capsules by adopting a lightweight model (student network) to mimic the teacher using the privileged knowledge. Such privileged information learned by the teacher contain similarities among capsules and are only available during the training stage of the student network. We evaluate the proposed architecture on two large-scale public EEG datasets, showing that our framework consistently enables student networks with different compression ratios to effectively learn from the teacher, even when provided with limited training samples. Lastly, our method achieves state-of-the-art results on one of the two datasets.

Beyond the Institution: An Informal Partnership Endeavored Toward Understanding and Addressing Black Teacher Recruitment and Retention.

Black teachers (BTs) are significantly underrepresented in the US teaching profession, yet there is still little focus on how to best hire, support, and retain them. This collaborative autoethnography documents our work in an urban characteristic school district and university in the southeastern US and how we leveraged our interpersonal and professional experiences with local educational and academic institutions to better understand the challenges associated with creating sustainable and systemic pathway and retention practices that prioritize BTs. We focus on the role of data and research in better understanding the localized experiences of BTs, share our vision for the Southeastern Black Teacher Network through an informal partnership, and offer recommendations for supporting BTs.

Distillation Sparsity Training Algorithm for Accelerating Convolutional Neural Networks in Embedded Systems

The rapid development of neural networks has come at the cost of increased computational complexity. Neural networks are both computationally intensive and memory intensive; as such, the minimal energy and computing power of satellites pose a challenge for automatic target recognition (ATR). Knowledge distillation (KD) can distill knowledge from a cumbersome teacher network to a lightweight student network, transferring the essential information learned by the teacher network. Thus, the concept of KD can be used to improve the accuracy of student networks. Even when learning from a teacher network, there is still redundancy in the student network. Traditional networks fix the structure before training, such that training does not improve the situation. This paper proposes a distillation sparsity training (DST) algorithm based on KD and network pruning to address the above limitations. We first improve the accuracy of the student network through KD, and then through network pruning, allowing the student network to learn which connections are essential. DST allows the teacher network to teach the pruned student network directly. The proposed algorithm was tested on the CIFAR-100, MSTAR, and FUSAR-Ship data sets, with a 50% sparsity setting. First, a new loss function for the teacher-pruned student was proposed, and the pruned student network showed a performance close to that of the teacher network. Second, a new sparsity model (uniformity half-pruning UHP) was designed to solve the problem that unstructured pruning does not facilitate the implementation of general-purpose hardware acceleration and storage. Compared with traditional unstructured pruning, UHP can double the speed of neural networks.

Semantic-aware knowledge distillation with parameter-free feature uniformization

Knowledge distillation aims to distill knowledge from teacher networks to train student networks. Distilling intermediate features has attracted much attention in recent years as it can be flexibly applied in various fields such as image classification, object detection and semantic segmentation. A critical obstacle of feature-based knowledge distillation is the dimension gap between the intermediate features of teacher and student, and plenty of methods have been proposed to resolve this problem. However, these works usually implement feature uniformization in an unsupervised way, lacking guidance to help the student network learn meaningful mapping functions in the uniformization process. Moreover, the dimension uniformization process of the student and teacher network is usually not equivalent as the mapping functions are different. To this end, some factors of the feature are discarded during parametric feature alignment, or some factors are blended in some non-parametric operations. In this paper, we propose a novel semantic-aware knowledge distillation scheme to solve these problems. We build a standalone feature-based classification branch to extract semantic-aware knowledge for better guiding the learning process of the student network. In addition, to avoid the inequivalence of feature uniformization between teacher and student, we design a novel parameter-free self-attention operation that can convert features of different dimensions into vectors of the same length. Experimental results show that the proposed knowledge distillation scheme outperforms existing feature-based distillation methods on the widely used CIFAR-100 and CINIC-10 datasets.

Customizing a teacher for feature distillation

Knowledge distillation is a method to train a lightweight network by transferring class probability knowledge from a cumbersome teacher network. However, transferring only the class probability knowledge would limit the distillation performance. Therefore, several approaches have been proposed to transfer the teacher's knowledge at the feature map level. In this paper, we revisit the feature distillation method and have found that the larger the teacher's architecture/capacity becomes, the more difficult it is for the student to imitate. Thus, the feature distillation method is unable to achieve its full potential. To address this, a novel end-to-end distillation framework, termed Customizing a Teacher for Feature Distillation (CTFD), is proposed to train a teacher to be more compatible with its student. In addition, we apply the customized teacher to three feature distillation methods. Moreover, data augmentation is used as a trick to train the student to improve its generalization performance. Extensive empirical experiments and analyses are conducted on three computer vision tasks, including image classification, transfer learning, and object detection, to substantiate the effectiveness of the proposed method.

DED: Diagnostic Evidence Distillation for acne severity grading on face images

Acne seriously affects people’s daily life. Acne severity level grading plays a decisive role in the cure. However, the acne criterion is not unified in the medical field. Most of the current studies explore the application of advanced visual models on acne severity grading but lack the adaptation to the characteristics of acne diagnosis. Meanwhile, some studies propose specifically adapted methods for respectively used acne criteria but cannot be used on other acne criteria. In this study, we propose an acne diagnosis method, Diagnostic Evidence Distillation (DED), that suitably adapts the characteristics of acne diagnosis and can be applied to diagnose under different acne criteria. Firstly, we fully investigate and analyze the commonality of various acne criteria and condense the face acne diagnosis into an unconventional image classification problem based on the diagnostic evidence of type and number of fine-grained lesions distributed on a whole face. Next, we propose the DED framework to adapt the characteristics of acne diagnosis. This framework uses the teacher–student structure of knowledge distillation methods to bring the diagnostic evidence, which is not available for new patients but only in training data, into the diagnosis model. To break the limitation of different criteria, the framework uses convolutional neural networks (CNNs) as backbones to imitate the global estimation of dermatology. We also propose a subtask joint learning for the teacher network to enhance its guidance to the student. The DED is applied to diagnose acne on two datasets ACNE04 and PLSBRACNE01 based on different mainstream acne criteria. The experimental results demonstrate that on both datasets, the DED effectively improves the diagnosis performance, exceeds the state-of-the-art and reaches the diagnostic level of dermatologists. The precision, sensitivity, specificity, Youden Index and accuracy reach 85.31%, 84.83%, 94.66%, 79.48% and 86.06% on the ACNE04 dataset and 69.16%, 65.62%, 88.93%, 54.54% and 67.56% on the PLSBRACNE01 dataset, respectively.

Pedagogical innovations in horticulture and landscape training: can we imagine an international network of innovative teachers?

Pedagogical innovations in horticulture and landscape training: can we imagine an international network of innovative teachers?

Pull & Push: Leveraging Differential Knowledge Distillation for Efficient Unsupervised Anomaly Detection and Localization

Recently, much attention has been paid to segmenting subtle unknown defect regions by knowledge distillation in an unsupervised setting. Most previous studies concentrated on guiding the student network to learn the same representations on the normality, neglecting the different behaviors of the abnormality. This leads to a high probability of false detection of subtle defects. To address such an issue, we propose to push representations on abnormal areas of the teacher and student network as far as possible while pulling representations on normal areas as close as possible. Based on this idea, we design an efficient teacher-student model for anomaly detection and localization, which maximizes pixel-wise discrepancies for anomalous regions approximated by data augmentation and simultaneously minimizes discrepancies for pixel-wise normal regions between these two networks. The explicit differential knowledge distillation enlarges the margin between normal representations and abnormal ones in favour of discriminating them. Then, the appropriate small student network is not only efficient, but more importantly, helps inhibit the generalization ability of anomalous patterns when learning normal patterns, facilitating the precise decision boundary. The experimental results on the MVTec AD, Fashion-MNIST, and CIFAR-10 datasets demonstrate that our proposed method achieves better performance than current state-of-the-art (SOTA) approaches. Especially, For the MVTec AD dataset with high resolution images, we achieve 98.1 AUROC% and 93.6 AUPRO% in anomaly localization, outperforming knowledge distillation based SOTA methods by 1.1 AUROC% and 1.5 AUPRO% with a lightweight model.

Knowledge Transfer Based Spatial Embedding Network for Plant Leaf Instance Segmentation

This paper proposes a method to segment plant leaves using knowledge distillation. Unlike the existing knowledge distillation method aimed at lightening the model, we use knowledge distillation to achieve good performance even with a small amount of dataset. Plants have many leaves, and each leaf is very small. Therefore, the leaf instance segmentation is performed based on spatial embedding. The teacher network is trained with a large dataset and then distills its segmentation knowledge into the student network. Two types of knowledge are distilled from the teacher network: attention distillation and region affinity distillation. The results of the experiment demonstrate that better instance segmentation can be achieved when knowledge distillation is used.

Modality Translation and Fusion for event-based semantic segmentation

Semantic segmentation is an important part of scene understanding in autonomous systems. Accurate segmentation results help the autonomous system better understand the surrounding environment, which improve the task success rate and enhance the system reliability. However, due to the hardware limitations of frame-based cameras, semantic segmentation based on RGB images under low-light conditions is still a difficult problem. The event camera, an emerging bio-inspired vision sensor, tries to fill these gaps with its high dynamic range. Most previous works resort to cross-modality knowledge distillation from a pretrained image-based teacher network to train an event-based network for semantic segmentation. However, a direct knowledge distillation is inadequate because of the different input modalities in the two networks. Utilizing image features to supervise event features cannot make full use of the knowledge in the teacher network. Thus, we propose a Modality Translation and Fusion (MTF) framework to distill diverse cross-modality knowledge. Specifically, we first develop a Modality Translation (MT) module to convert events into image modality. With different input modalities, two feature extractors are built to learn diverse and complementary knowledge from the teacher network. Then, to better utilize the knowledge, we propose a Residual-based Coordinate Attention Fusion (RCAF) module to fuse the multi-scale features from different modalities. Finally, extensive experiments show that our MTF is superior to state-of-the-art (SOTA) approaches.

Effects of the visual analytics of peer feedback on teachers’ TPACK development in a lesson study

ABSTRACT Teacher networks and communities have played an important role in teacher professional development. In such contexts, teachers often receive extensive feedback from peers as part of social learning. However, many teachers have difficulty identifying essential information from a large amount of peer feedback, which may impede self-reflection and peer learning. This study proposes the use of computer-assisted visual analytics of peer feedback to address this challenge. A visualisation-based analytical dashboard was designed and applied to help teachers analyse and reflect on peer feedback in a lesson study community for Technological Pedagogical Content Knowledge (TPACK) development. Primary school teachers participated in the lesson study, in which they collaborated to discuss lesson plans, observe recorded lessons, and give peer ratings and comments using an online platform. By comparing the performance between those using the analytical dashboard and others not using it, the results show that the approach has promising effects on improving teachers’ TPACK as reflected in the lesson plans and their perceived confidence in TPACK. The implications of the findings are also discussed.