Teacher Network Research Articles

Exploring Generalizable Distillation for Efficient Medical Image Segmentation.

Efficient medical image segmentation aims to provide accurate pixel-wise predictions with a lightweight implementation framework. However, existing lightweight networks generally overlook the generalizability of the cross-domain medical segmentation tasks. In this paper, we propose Generalizable Knowledge Distillation (GKD), a novel framework for enhancing the performance of lightweight networks on cross-domain medical segmentation by generalizable knowledge distillation from powerful teacher networks. Considering the domain gaps between different medical datasets, we propose the Model-Specific Alignment Networks (MSAN) to obtain the domain-invariant representations. Meanwhile, a customized Alignment Consistency Training (ACT) strategy is designed to promote the MSAN training. Based on the domain-invariant vectors in MSAN, we propose two generalizable distillation schemes, Dual Contrastive Graph Distillation (DCGD) and Domain-Invariant Cross Distillation (DICD). In DCGD, two implicit contrastive graphs are designed to model the intra-coupling and inter-coupling semantic correlations. Then, in DICD, the domain-invariant semantic vectors are reconstructed from two networks (i.e., teacher and student) with a crossover manner to achieve simultaneous generalization of lightweight networks, hierarchically. Moreover, a metric named Fréchet Semantic Distance (FSD) is tailored to verify the effectiveness of the regularized domain-invariant features. Extensive experiments conducted on the Liver, Retinal Vessel and Colonoscopy segmentation datasets demonstrate the superiority of our method, in terms of performance and generalization ability on lightweight networks.

UAWC: An intelligent underwater acoustic target recognition system for working conditions mismatching

Underwater acoustic target recognition (UATR) systems are crucial to both military and civilian activities. However, the complex ship working conditions will largely affect the performance of recognition systems, especially in the case of working conditions mismatching (WCMM). For WCMM problems, an intelligent UATR system for working condition mismatching (UAWC) is proposed. UAWC uses auditory features as input to the system and uses knowledge distillation to learn the intrinsic connections of target features under different working conditions. In the proposed approach, the teacher network obtains initial knowledge by utilizing a large amount of existing working condition data. Next, the student network uses a small amount of target working data for training, and extracts incremental knowledge in teacher network through knowledge distillation technology to enhance the accuracy of its classification of target working data, so as to effectively deal with the WCMM problem.The tests make use of datasets for ship-radiated noise under various working conditions.The results showed that UAWC performs better than other methods on a wide range of WCMM problems.

An adaptive semi-supervised self-learning method for online state of health estimation of lithium-ion batteries

Accurate and precise online estimation of the state of health (SOH) is crucial when managing lithium-ion batteries. Most existing SOH estimation methods rely on supervised learning algorithms utilizing large amounts of labeled data. However, lithium-ion batteries are typically operated under dynamic conditions, including significant amounts of unlabeled charging or discharging data in online application scenarios. To fully utilize these data, we propose an adaptive semi-supervised self-learning teacher-student model (AS3LTSM) method for online SOH estimation. First, four physically interpretable health indicators (PIHIs) are extracted from the voltage and current data. The Pearson correlation coefficient (PCC) is used to assess significant associations between PIHIs and the SOH. Regressive and autoregressive long short-term memory (LSTM) models are selected as the teacher and student networks. Knowledge is transferred from the teacher to the student through pseudolabels, which guide the updating and evolution of the student network. Furthermore, a self-learning strategy and a retraining process for improving the long-term estimation accuracy are proposed. Two public datasets are used for comparison and ablation experiments. Experimental analysis validates the improved effectiveness and performance of the proposed method, with the RMSE and MAPE of the three experimental groups all within 1.3 % and 1.29 %, respectively.

Examining the resilience of two early career world language teachers: a two-year case study

ABSTRACT Language teachers’ resilience, or their socially-mediated capacity to adapt to adverse conditions and thrive, has garnered attention amidst evolving challenges to language education and bilingual teachers in diverse spaces. Taking an ecological-collective perspective towards language teacher resilience, this qualitative case study examined the experiences of two early career Spanish world language teachers in the United States, paying particular attention to the personal and contextual resources they leveraged for their resilience during and after their transition into teaching. Through a series of semi-structured interviews, three themes were identified representing the varied purposes for which the Spanish teachers mobilised resources for their resilience at different times: planning for resilience in advance during the job search, maintaining resilience amidst initial struggles, and reestablishing resilience. Findings show that the types of resources used by the early career world language teachers shifted over time and based on need, the most salient of which included their knowledge of their teaching context, their agency and persistence, support and autonomy from their school administrators, and connections with networks of other language teachers. Implications for teacher education programmes, language teacher resilience, and future research are discussed.

CloudSeg: A multi-modal learning framework for robust land cover mapping under cloudy conditions

Cloud coverage poses a significant challenge to optical image interpretation, degrading ground information on Earth’s surface. Synthetic aperture radar (SAR), with its ability to penetrate clouds, provides supplementary information to optical data. However, existing optical-SAR fusion methods predominantly focus on cloud-free scenarios, neglecting the practical challenge of semantic segmentation under cloudy conditions. To tackle this issue, we propose CloudSeg, a novel framework tailored for land cover mapping in the presence of clouds. It addresses the challenges posed by cloud cover from two aspects: reducing semantic ambiguity in areas of the cloudy image that are obscured by clouds and enhancing effective information in the unobstructed portions. Specifically, CloudSeg employs a multi-task learning strategy to simultaneously handle low-level visual task and high-level semantic understanding task, mitigating the semantic ambiguity caused by cloud cover by acquiring discriminative features through an auxiliary cloud removal task. Additionally, CloudSeg incorporates a knowledge distillation strategy, which utilizes the knowledge learned by the teacher network under cloud-free conditions to guide the student network to overcome the interference of cloud-covered areas, enhancing the valuable information from the unobstructed parts of cloud-covered images. Extensive experiments conducted on two datasets, M3M-CR and WHU-OPT-SAR, demonstrate the effectiveness and superiority of the proposed CloudSeg method for land cover mapping under cloudy conditions. Specifically, CloudSeg outperforms the state-of-the-art competitors by 3.16% in terms of mIoU on M3M-CR and by 5.56% on WHU-OPT-SAR, highlighting its substantial advantages for analyzing regions frequently obscured by clouds. Codes are available at https://github.com/xufangchn/CloudSeg.

Leveraging bone age assessment via a novel joint decomposition teacher–student learning paradigm from X-ray images

ABSTRACT Advancements in technology has accelerated the evolution of bone age assessment (BAA) methodologies, one of which is deep learning algorithms, which overcome the drawbacks of conventional approaches. In spite of excellent effectiveness of deep neural networks in detection of the correct class for bone age, they have a significant degree of complexity due to the numerous parameters they employ for each region of interest (ROI). In this paper, we propose a BAA method using a hybrid knowledge distillation (KD) paradigm in order to conquer this difficulty by mapping different ROIs into a single ROI. In this regard, the student receives knowledge from a teacher network that has been pre-trained on six ROIs including bones of five fingers and the wrist, transfers the knowledge of its final response layer and internal layers to the student. Then, six student models each of which is constructed based on just one of these ROIs, while receiving the information of the teacher model. Empirical results on digital hand atlas report that our student model trained on one ROI obtains 95% accuracy on 19 classes of bone age.

Decoding visual brain representations from electroencephalography through knowledge distillation and latent diffusion models

Decoding visual representations from human brain activity has emerged as a thriving research domain, particularly in the context of brain–computer interfaces. Our study presents an innovative method that employs knowledge distillation to train an EEG classifier and reconstruct images from the ImageNet and THINGS-EEG 2 datasets using only electroencephalography (EEG) data from participants who have viewed the images themselves (i.e. “brain decoding”). We analyzed EEG recordings from 6 participants for the ImageNet dataset and 10 for the THINGS-EEG 2 dataset, exposed to images spanning unique semantic categories. These EEG readings were converted into spectrograms, which were then used to train a convolutional neural network (CNN), integrated with a knowledge distillation procedure based on a pre-trained Contrastive Language-Image Pre-Training (CLIP)-based image classification teacher network. This strategy allowed our model to attain a top-5 accuracy of 87%, significantly outperforming a standard CNN and various RNN-based benchmarks. Additionally, we incorporated an image reconstruction mechanism based on pre-trained latent diffusion models, which allowed us to generate an estimate of the images that had elicited EEG activity. Therefore, our architecture not only decodes images from neural activity but also offers a credible image reconstruction from EEG only, paving the way for, e.g., swift, individualized feedback experiments.

Learning From Human Educational Wisdom: A Student-Centered Knowledge Distillation Method.

Existing studies on knowledge distillation typically focus on teacher-centered methods, in which the teacher network is trained according to its own standards before transferring the learned knowledge to a student one. However, due to differences in network structure between the teacher and the student, the knowledge learned by the former may not be desired by the latter. Inspired by human educational wisdom, this paper proposes a Student-Centered Distillation (SCD) method that enables the teacher network to adjust its knowledge transfer according to the student network's needs. We implemented SCD based on various human educational wisdom, e.g., the teacher network identified and learned the knowledge desired by the student network on the validation set, and then transferred it to the latter through the training set. To address the problems of current deficiency knowledge, hard sample learning and knowledge forgetting faced by a student network in the learning process, we introduce and improve Proportional-Integral-Derivative (PID) algorithms from automation fields to make them effective in identifying the current knowledge required by the student network. Furthermore, we propose a curriculum learning-based fuzzy strategy and apply it to the proposed PID control algorithm, such that the student network in SCD can actively pay attention to the learning of challenging samples after with certain knowledge. The overall performance of SCD is verified in multiple tasks by comparing it with state-of-the-art ones. Experimental results show that our student-centered distillation method outperforms existing teacher-centered ones.

Knowledge distillation of face recognition via attention cosine similarity review

AbstractDeep learning‐based face recognition models have demonstrated remarkable performance in benchmark tests, and knowledge distillation technology has been frequently accustomed to obtain high‐precision real‐time face recognition models specifically designed for mobile and embedded devices. However, in recent years, the knowledge distillation methods for face recognition, which mainly focus on feature or logit knowledge distillation techniques, neglect the attention mechanism that play an important role in the domain of neural networks. An innovation cross‐stage connection review path of the attention cosine similarity knowledge distillation method that unites the attention mechanism with review knowledge distillation method is proposed. This method transfers the attention map obtained from the teacher network to the student through a cross‐stage connection path. The efficacy and excellence of the proposed algorithm are demonstrated in popular benchmark tests.

Multiscale knowledge distillation with attention based fusion for robust human activity recognition

Knowledge distillation is an effective approach for training robust multi-modal machine learning models when synchronous multimodal data are unavailable. However, traditional knowledge distillation techniques have limitations in comprehensively transferring knowledge across modalities and models. This paper proposes a multiscale knowledge distillation framework to address these limitations. Specifically, we introduce a multiscale semantic graph mapping (SGM) loss function to enable more comprehensive knowledge transfer between teacher and student networks at multiple feature scales. We also design a fusion and tuning (FT) module to fully utilize correlations within and between different data types of the same modality when training teacher networks. Furthermore, we adopt transformer-based backbones to improve feature learning compared to traditional convolutional neural networks. We apply the proposed techniques to multimodal human activity recognition and compared with the baseline method, it improved by 2.31% and 0.29% on the MMAct and UTD-MHAD datasets. Ablation studies validate the necessity of each component.

DistilIQA: Distilling Vision Transformers for no-reference perceptual CT image quality assessment

No-reference image quality assessment (IQA) is a critical step in medical image analysis, with the objective of predicting perceptual image quality without the need for a pristine reference image. The application of no-reference IQA to CT scans is valuable in providing an automated and objective approach to assessing scan quality, optimizing radiation dose, and improving overall healthcare efficiency. In this paper, we introduce DistilIQA, a novel distilled Vision Transformer network designed for no-reference CT image quality assessment. DistilIQA integrates convolutional operations and multi-head self-attention mechanisms by incorporating a powerful convolutional stem at the beginning of the traditional ViT network. Additionally, we present a two-step distillation methodology aimed at improving network performance and efficiency. In the initial step, a ”teacher ensemble network” is constructed by training five vision Transformer networks using a five-fold division schema. In the second step, a ”student network”, comprising of a single Vision Transformer, is trained using the original labeled dataset and the predictions generated by the teacher network as new labels. DistilIQA is evaluated in the task of quality score prediction from low-dose chest CT scans obtained from the LDCT and Projection data of the Cancer Imaging Archive, along with low-dose abdominal CT images from the LDCTIQAC2023 Grand Challenge. Our results demonstrate DistilIQA‘s remarkable performance in both benchmarks, surpassing the capabilities of various CNNs and Transformer architectures. Moreover, our comprehensive experimental analysis demonstrates the effectiveness of incorporating convolutional operations within the ViT architecture and highlights the advantages of our distillation methodology.

Enhanced Attention Guided Teacher–Student Network for Weakly Supervised Object Detection

Weakly Supervised Object Detection (WSOD) has attracted increasing attention due to the convenience and low-cost of acquiring image-level annotations. Most existing WSOD methods follow Multiple Instance Learning (MIL) paradigm to select bounding boxes from proposals based on their classification scores. However, MIL-based WSOD methods often focus on discriminative regions, leading to incomplete and missing instances. To address these issues, we introduce a novel WSOD framework named Enhanced Attention Guided Teacher–Student Network (EATSN). This framework aims to improve detection performance through the guidance of attention map and consistency learning. specifically, we initially train a teacher network using MIL process to generate pseudo ground-truth labels. Subsequently, the weak augmented and strong augmented images are fed into teacher and student models to produce the enhanced attention map. During the training iterations, pseudo labels are utilized to guide the student model, while the teacher model refines its parameters through the Exponential Moving Average(EMA) from the student. Finally we design a proposal selection method that leverages the enhanced attention map and bounding boxes scores to achieve better detection results. Experimental results on benchmark datasets demonstrate that our method achieves comparable performance.

Self-Knowledge Distillation via Progressive Associative Learning

As a specific form of knowledge distillation (KD), self-knowledge distillation enables a student network to progressively distill its own knowledge without relying on a pretrained, complex teacher network; however, recent studies of self-KD have discovered that additional dark knowledge captured by auxiliary architecture or data augmentation could create better soft targets for enhancing the network but at the cost of significantly more computations and/or parameters. Moreover, most existing self-KD methods extract the soft label as a supervisory signal from individual input samples, which overlooks the knowledge of relationships among categories. Inspired by human associative learning, we propose a simple yet effective self-KD method named associative learning for self-distillation (ALSD), which progressively distills richer knowledge regarding the relationships between categories across independent samples. Specifically, in the process of distillation, the propagation of knowledge is weighted based on the intersample relationship between associated samples generated in different minibatches, which are progressively estimated with the current network. In this way, our ALSD framework achieves knowledge ensembling progressively across multiple samples using a single network, resulting in minimal computational and memory overhead compared to existing ensembling methods. Extensive experiments demonstrate that our ALSD method consistently boosts the classification performance of various architectures on multiple datasets. Notably, ALSD pushes forward the self-KD performance to 80.10% on CIFAR-100, which exceeds the standard backpropagation by 4.81%. Furthermore, we observe that the proposed method shows comparable performance with the state-of-the-art knowledge distillation methods without the pretrained teacher network.

Semi-supervised correction model for turbulence-distorted images

Significant progress has been made in addressing turbulence distortion in recent years, but persistent challenges remain. Firstly, existing methods heavily rely on fully supervised optimization strategies and synthetic datasets, posing difficulties in effectively utilizing unlabeled real data for training. Secondly, most approaches construct networks in a straightforward manner, overlooking the representation model of phase distortion and point spread function (PSF) in spatial and channel dimensions. This oversight restricts the potential for distortion correction. To address these challenges, this paper proposes a semi-supervised atmospheric turbulence correction method based on the mean-teacher framework. Our approach imposes constraints on the unlabeled data of student networks using pseudo-labels generated by teacher networks, thereby enhancing the generalization ability by leveraging information from unlabeled data. Furthermore, we introduce to use no-reference image quality assessment criterion to select the most reliable pseudo-label for each unlabeled sample by predicting physical parameters that indicating the level of degradation. Additionally, we propose to combine sliding window-based self-attention with channel attention to facilitate local-global context interaction. This design is inspired by the representation of phase distortion and PSF, which can be characterized by coefficients and basis functions corresponding to the channel-wise representation of convolutional neural network features. Moreover, the base functions exhibit spatial correlation, akin to Zenike and Airy disks. Experimental results show that the proposed method surpasses state-of-the-art models.

Semi-supervised CT image segmentation via contrastive learning based on entropy constraints.

Deep learning-based methods for fast target segmentation of computed tomography (CT) imaging have become increasingly popular. The success of current deep learning methods usually depends on a large amount of labeled data. Labeling medical data is a time-consuming and laborious task. Therefore, this paper aims to enhance the segmentation of CT images by using a semi-supervised learning method. In order to utilize the valid information in unlabeled data, we design a semi-supervised network model for contrastive learning based on entropy constraints. We use CNN and Transformer to capture the image's local and global feature information, respectively. In addition, the pseudo-labels generated by the teacher networks are unreliable and will lead to degradation of the model performance if they are directly added to the training. Therefore, unreliable samples with high entropy values are discarded to avoid the model extracting the wrong features. In the student network, we also introduce the residual squeeze and excitation module to learn the connection between different channels of each layer feature to obtain better segmentation performance. We demonstrate the effectiveness of the proposed method on the COVID-19 CT public dataset. We mainly considered three evaluation metrics: DSC, HD95, and JC. Compared with several existing state-of-the-art semi-supervised methods, our method improves DSC by 2.3%, JC by 2.5%, and reduces HD95 by 1.9mm.In this paper, a semi-supervised medical image segmentation method is designed by fusing CNN and Transformer and utilizing entropy-constrained contrastive learning loss, which improves the utilization of unlabeled medical images.

The Critical Roles of a Mathematics Specialist in Establishing Effective, Coordinated Professional Development Systems

ABSTRACT For nearly three decades, mathematics research has indicated the need for high-quality mathematics instruction including both ambitious and equitable practices, as well as continuous, innovative learning opportunities for educators. Teaching practices and standards for mathematical practice have been identified to provide guidance. While research connects effective professional development (PD) to improved instructional practices, there is a need for additional research on the role mathematics specialists play in developing and sustaining PDs. This qualitative case study examined the multiple roles of a specialist while laying the foundational components for a coordinated PD system to improve and strengthen mathematics teaching and learning in elementary classrooms. The PD efforts included the implementation of a teacher subsystem where a specialist led all components consisting of pull-out PD, mathematics coaching, collaborative time, and teacher networks. This study took place at a rural, pre-K through fourth-grade school in a southeastern state. Analysis of the data identified the specialist’s key roles in each component of the teacher subsystem. The results indicated that the development and implementation of a teacher subsystem impacts the overall effectiveness of PD. Findings serve as a foundation for specialists to design and implement coordinated efforts that can have positive impacts on the teaching and learning of mathematics.

ADNet: A Real-Time Floating Algae Segmentation Using Distillation Network

The segmentation of floating algae is a hot topic in the field of marine environmental research. Given the vastness of coastal areas and complex environments, algae detection models must have both higher performance and lower deployment costs. However, relying solely on a single Convolutional Neural Network (CNN) or transformer structure fails to achieve this objective. In this paper, a novel real-time floating algae segmentation method using a distillation network (ADNet) is proposed, based on the RGB images. ADNet can effectively transfer the performance of the transformer-based teacher network to the CNN-based student model while preserving its lightweight design. Faced with complex marine environments, we introduce a novel Channel Purification Module (CPM) to simultaneously strengthen algae features and purify interference responses. Importantly, the CPM achieves this operation without increasing any learnable parameters. Moreover, considering the huge scale differences among algae targets in surveillance RGB images, we propose a lightweight multi-scale feature fusion network (L-MsFFN) to improve the student’s modeling ability across various scales. Additionally, to mitigate interference from low-level noises on higher-level semantics, a novel position purification module (PPM) is proposed. The PPM can achieve more accurate weight attention calculation between different pyramid levels, thereby enhancing the effectiveness of fusion. Compared to CNNs and transformers, our ADNet strikes an optimal balance between performance and speed. Extensive experimental results demonstrate that our ADNet achieves higher application performance in the field of floating algae monitoring tasks.

Robust anomaly detection in industrial images by blending global–local features

AbstractIndustrial image anomaly detection achieves automated detection and localization of defects or abnormal regions in images through image processing and deep learning techniques. Currently, utilizing the approach of reverse knowledge distillation has yielded favourable outcomes. However, it is still a challenge in terms of the feature extraction capability of the image and the robustness of the decoding of the student network. This study first addresses the issue that the teacher network has not been able to extract global information more effectively. To acquire more global information, a vision transformer network is introduced to enhance the model's global information extraction capability, obtaining better features to further assist the student network in decoding. Second, for anomalous samples, to address the low similarity between features extracted by the teacher network and features restored by the student network, Gaussian noise is introduced. This further increases the probability that the features decoded by the student model match normal sample features, enhancing the robustness of the student model. Extensive experiments were conducted on industrial image datasets AeBAD, MvtecAD, and BTAD. In the AeBAD dataset, under the PRO performance metric, the result is 89.83%, achieving state‐of‐the‐art performance. Under the AUROC performance metric, it reaches 83.35%. Similarly, good results were achieved on the MvtecAD and BTAD datasets. The proposed method's effectiveness and performance advantages were validated across multiple industrial datasets, providing a valuable reference for the application of industrial image anomaly detection methods.

LPI radar waveform recognition based on semi-supervised model all mean teacher

Low probability of intercept (LPI) radar signal identification plays an important role in electronic warfare, but most existing algorithms are proposed under the condition of sufficient samples, ignoring the problem of a small amount of labeled data in the actual electromagnetic environment. To solve the problem, in this paper, a semi-supervised learning model All Mean Teacher (AMT) based on Mean Teacher (MT) is proposed. First, the LPI radar signal is transformed into Time-frequency images (TFIs) by using the Choi-Williams distribution, and Random Erasing is used for TFIs which improves the generalization ability of the model. Then the Multi-headed Self-Attention Network (MSA-Net) is aimed to extract features, combined with AMT to realize the automatic waveform recognition of radar signals. MSA-Net facilitates feature information propagation by computing contrast costs on TFIs between the student and teacher networks. It solves the problem that TFIs are not easy to train for small amounts of labeled data, improving the accuracy of signal recognition in semi-supervised learning scenarios. Experimental results show that the average recognition accuracy of the proposed method is up to 85.7% at a signal-to-noise ratio of -8 dB.

Research on disease diagnosis based on teacher-student network and Raman spectroscopy.

Diabetic nephropathy is a serious complication of diabetes, and primary Sjögren's syndrome is a disease that poses a major threat to women's health. Therefore, studying these two diseases is of practical significance. In the field of spectral analysis, although common Raman spectral feature selection models can effectively extract features, they have the problem of changing the characteristics of the original data. The teacher-student network combined with Raman spectroscopy can perform feature selection while retaining the original features, and transfer the performance of the complex deep neural network structure to another lightweight network structure model. This study selects five flow learning models as the teacher network, builds a neural network as the student network, uses multi-layer perceptron for classification, and selects the optimal features based on the evaluation indicators accuracy, precision, recall, and F1-score. After five-fold cross-validation, the research results show that in the diagnosis of diabetic nephropathy, the optimal accuracy rate can reach 98.3%, which is 14.02% higher than the existing research; in the diagnosis of primary Sjögren's syndrome, the optimal accuracy rate can be reached 100%, which is 10.48% higher than the existing research. This study proved the feasibility of Raman spectroscopy combined with teacher-student network in the field of disease diagnosis by producing good experimental results in the diagnosis of diabetic nephropathy and primary Sjögren's syndrome.