Teacher-student Network Research Articles

A MLP architecture fusing RGB and CASSI for computational spectral imaging

The coded Aperture Snapshot Spectral Imaging (CASSI) system offers significant advantages in dynamically acquiring hyper-spectral images compared to traditional measurement methods. However, it faces the following challenges: (1) Traditional masks rely on random patterns or analytical design, limiting CASSI’s performance improvement. (2) Existing CASSI reconstruction algorithms do not fully utilize RGB information. (3) High-quality reconstruction algorithms are often slow and limited to offline scene reconstruction. To address these issues, this paper proposes a new MLP architecture, Spectral–Spatial MLP (SSMLP), which replaces the transformer structure with a network using CASSI measurements and RGB as multimodal inputs. This maintains reconstruction quality while significantly improving reconstruction speed. Additionally, we constructed a teacher-student network (SSMLP with a teacher, SSMLP-WT) to transfer the knowledge learned from a large model to a smaller network, further enhancing the smaller network’s accuracy. Extensive experiments show that SSMLP matches the performance of transformer-based structures in spectral image reconstruction while improving inference speed by at least 50%. The reconstruction quality of SSMLP-WT is further improved by knowledge transfer without changing the network, and the teacher boosts the performance by 0.92 dB (44.73 dB vs. 43.81 dB).

Test time adaptation for cross-domain sleep stage classification

Domain shift is a serious problem with existing cross-domain sleep stage classification methods. However, most existing domain adaptation methods require access to a large amount of source domain data containing patient information, which may lead to infringement of patient privacy. At the same time, existing methods require collecting all information in advance when processing target data, which cannot meet the situation where test data arrives in batches or in order in actual scenarios. To this end, we propose a test time adaptation framework based on knowledge distillation for cross-domain sleep stage classification. This teacher-student network framework uses pre-trained model to learn variant features and robust representations. We evaluate our framework in the cross domain scenario composed of three datasets and validate its superiority, real time performance and privacy protection capability.

Enhancing Precision in Cardiac Segmentation for Magnetic Resonance-Guided Radiation Therapy Through Deep Learning

IntroductionCardiac substructure dose metrics are more strongly linked to late cardiac morbidities than whole-heart metrics. MR-guided radiation therapy (MRgRT) enables substructure visualization during daily localization, allowing potential for enhanced cardiac sparing. We extend a publicly available state-of-the-art deep learning (DL) framework, nnU-Net, to incorporate self-distillation (nnU-Net.wSD) for substructure segmentation for MRgRT. MethodsEighteen (Institute A) patients who underwent thoracic or abdominal radiation therapy on a 0.35 T MR-guided linac were retrospectively evaluated. On each image, one of two radiation oncologists delineated reference contours of 12 cardiac substructures (chambers, great vessels, and coronary arteries) used to train (n=10), validate (n=3), and test (n=5) nnU-Net.wSD leveraging a teacher-student network and comparing to standard 3D U-Net. The impact of using simulation data or including 3-4 daily images for augmentation during training was evaluated for nnU-Net.wSD. Geometric metrics (Dice similarity coefficient (DSC), mean distance to agreement (MDA), and 95% Hausdorff distance (HD95)), visual inspection, and clinical dose volume histograms (DVHs) were evaluated. To determine generalizability, Institute A's model was tested on an unlabeled dataset from Institute B (n=22) and evaluated via consensus scoring and volume comparisons. ResultsnnU-Net.wSD yielded a DSC (reported mean ± standard deviation) of 0.65±0.25 across the 12 substructures (Chambers: 0.85±0.05, Great Vessels: 0.67±0.19, and Coronary Arteries 0.33±0.16, mean MDA <3 mm, and mean HD95 <9 mm) while outperforming the 3D U-Net (0.583±0.28, p<0.01). Leveraging fractionated data for augmentation improved over a single MR-SIM timepoint (0.579±0.29, p<0.01). Predicted contours yielded DVHs that closely matched the clinical treatment plans where mean and D0.03cc doses deviated by 0.32±0.5 Gy and 1.42±2.6 Gy respectively. No statistically significant differences between Institute A and B volumes (p>0.05) for 11 of 12 substructures with larger volumes requiring minor changes and coronary arteries exhibiting more variability. ConclusionsThis work is a critical step to rapid and reliable cardiac substructure segmentation to improve cardiac sparing in low-field MRgRT.

Research on disease diagnosis based on teacher-student network and Raman spectroscopy.

Diabetic nephropathy is a serious complication of diabetes, and primary Sjögren's syndrome is a disease that poses a major threat to women's health. Therefore, studying these two diseases is of practical significance. In the field of spectral analysis, although common Raman spectral feature selection models can effectively extract features, they have the problem of changing the characteristics of the original data. The teacher-student network combined with Raman spectroscopy can perform feature selection while retaining the original features, and transfer the performance of the complex deep neural network structure to another lightweight network structure model. This study selects five flow learning models as the teacher network, builds a neural network as the student network, uses multi-layer perceptron for classification, and selects the optimal features based on the evaluation indicators accuracy, precision, recall, and F1-score. After five-fold cross-validation, the research results show that in the diagnosis of diabetic nephropathy, the optimal accuracy rate can reach 98.3%, which is 14.02% higher than the existing research; in the diagnosis of primary Sjögren's syndrome, the optimal accuracy rate can be reached 100%, which is 10.48% higher than the existing research. This study proved the feasibility of Raman spectroscopy combined with teacher-student network in the field of disease diagnosis by producing good experimental results in the diagnosis of diabetic nephropathy and primary Sjögren's syndrome.

Dual knowledge distillation for visual tracking with teacher–student network

Dual knowledge distillation for visual tracking with teacher–student network

Shared Knowledge Distillation Network for Object Detection

Object detection based on Knowledge Distillation can enhance the capabilities and performance of 5G and 6G networks in various domains, such as autonomous vehicles, smart surveillance, and augmented reality. The integration of object detection with Knowledge Distillation techniques is expected to play a pivotal role in realizing the full potential of these networks. This study presents Shared Knowledge Distillation (Shared-KD) as a solution to overcome optimization challenges caused by disparities in cross-layer features between teacher–student networks. The significant gaps in intermediate-level features between teachers and students present a considerable obstacle to the efficacy of distillation. To tackle this issue, we draw inspiration from collaborative learning in real-world education, where teachers work together to prepare lessons and students engage in peer learning. Building upon this concept, our innovative contributions in model construction are highlighted as follows: (1) A teacher knowledge augmentation module: this module is proposed to combine lower-level teacher features, facilitating the knowledge transfer from the teacher to the student. (2) A student mutual learning module is introduced to enable students to learn from each other, mimicking the peer learning concept in collaborative learning. (3) The Teacher Share Module combines lower-level teacher features: the specific functionality of the teacher knowledge augmentation module is described, which involves combining lower-level teacher features. (4) The multi-step transfer process can be easily optimized due to the minimal gap between the features: the proposed approach breaks down the knowledge transfer process into multiple steps, which can be easily optimized due to the minimal gap between the features involved in each step. Shared-KD uses simple feature losses without additional weights in transformation, resulting in an efficient distillation process that can be easily combined with other methods for further improvement. The effectiveness of our approach is validated through experiments on popular tasks such as object detection and instance segmentation.

Attribute discrimination combined with selected sample dropout for unsupervised domain adaptive person re-identification

The popular clustering-based Unsupervised Domain Adaptive (UDA) person re-identification (re-ID) does not require additional annotation. However, owing to unsatisfactory feature embedding and imperfect clustering, most existing clustering-based methods suffer from the noise of pseudo labels in the target domain, which will lead to a serious performance degradation. To reduce the negative impact of noisy pseudo labels on training, we put forward an approach named selected sample dropout (SSD) in the training stage, which defines a criterion for evaluating the noise level of pseudo labels. SSD would mine and discard samples with noisy pseudo labels before training, and then all the remaining samples are fed into the network to train. On top of a strong baseline, SSD is proved to be effective. We call the baseline of adding SSD as the selected sample dropout-enhanced teacher-student network (SSD-TSNet). In addition, considering the robustness of pedestrian gender, we use it as auxiliary information in the SSD-TSNet test stage. Specifically, we propose a pedestrian gender attribute discriminator (GAD) to predict gender labels. Based on predictive gender labels, SSD-TSNet could retrieve one person among other persons with the same gender as the person, thus narrowing the search space of re-ID. The proposed SSD-TSNet and GAD are integrated into one framework, and extensive experiments on four widely used UDA benchmark protocols demonstrate its competitive performance. Specifically, our method outperforms the baseline by 11.6% mAP on the Duke-to-Market task, while surpassing the state-of-the-art method by 0.5% mAP on the Market-to-Duke task.

Deconfounded Cross-modal Matching for Content-based Micro-video Background Music Recommendation

Object-oriented micro-video background music recommendation is a complicated task where the matching degree between videos and background music is a major issue. However, music selections in user-generated content (UGC) are prone to selection bias caused by historical preferences of uploaders. Since historical preferences are not fully reliable and may reflect obsolete behaviors, over-reliance on them should be avoided as knowledge and interests dynamically evolve. In this article, we propose a Deconfounded Cross-Modal matching model to mitigate such bias. Specifically, uploaders’ personal preferences of music genres are identified as confounders that spuriously correlate music embeddings and background music selections, causing the learned system to over-recommend music from majority groups. To resolve such confounders, backdoor adjustment is utilized to deconfound the spurious correlation between music embeddings and prediction scores. We further utilize Monte Carlo estimator with batch-level average as the approximations to avoid integrating the entire confounder space calculated by the adjustment. Furthermore, we design a teacher–student network to utilize the matching of music videos, which is professionally generated content (PGC) with specialized matching, to better recommend content-matching background music. The PGC data are modeled by a teacher network to guide the matching of uploader-selected UGC data of student network by Kullback–Leibler–based knowledge transfer. Extensive experiments on the TT-150k-genre dataset demonstrate the effectiveness of the proposed method. The code is publicly available on https://github.com/jing-1/DecCM

Lifelong Dual Generative Adversarial Nets Learning in Tandem.

Continually capturing novel concepts without forgetting is one of the most critical functions sought for in artificial intelligence systems. However, even the most advanced deep learning networks are prone to quickly forgetting previously learned knowledge after training with new data. The proposed lifelong dual generative adversarial networks (LD-GANs) consist of two generative adversarial networks (GANs), namely, a Teacher and an Assistant teaching each other in tandem while successively learning a series of tasks. A single discriminator is used to decide the realism of generated images by the dual GANs. A new training algorithm, called the lifelong self knowledge distillation (LSKD) is proposed for training the LD-GAN while learning each new task during lifelong learning (LLL). LSKD enables the transfer of knowledge from one more knowledgeable player to the other jointly with learning the information from a newly given dataset, within an adversarial playing game setting. In contrast to other LLL models, LD-GANs are memory efficient and does not require freezing any parameters after learning each given task. Furthermore, we extend the LD-GANs to being the Teacher module in a Teacher-Student network for assimilating data representations across several domains during LLL. Experimental results indicate a better performance for the proposed framework in unsupervised lifelong representation learning when compared to other methods.

Knowledge-integrated Multi-modal Movie Turning Point Identification

The rapid development of artificial intelligence provides rich technologies and tools for the automated understanding of literary works. As a comprehensive carrier of storylines, movies are natural multimodal data sources that provide sufficient data foundations, and how to fully leverage the benefits of data remains a sustainable research hotspot. In addition, the efficient representation of multi-source data also poses new challenges for information fusion technology. Therefore, we propose a knowledge-enhanced turning points identification (KTPi) method for multimodal scene recognition. First, the BiLSTM method is used to encode scene text and integrate contextual information into scene representations to complete text sequence modeling. Then, the graph structure is used to model all scenes, which strengthens long-range semantic dependencies between scenes and enhances scene representations using graph convolution network. After, the self-supervised method is used to obtain the optimal number of neighboring nodes in sparse graph. Next, actor and verb knowledge involved in the scene text are added to the multimodal data to enhance the diversity of scene feature expressions. Finally, the teacher-student network strategy is used to train the KTPi model. Experimental results show that KTPi outperforms baseline methods in scene role recognition tasks, and ablation experiments show that incorporating knowledge into multimodal model can improve its performance.

A progressive distillation network for practical image-based virtual try-on

The image-based virtual try-on technology aims to match in-store clothing to person's image wearing clothes. Previous methods require a large amount of input information for each try-on result, which is not practical, and slight deviations in input information can cause a large number of artifacts in the try-on results. In recent years, researchers have paid attention to parsing-free virtual try-on methods, and a groundbreaking work used knowledge distillation techniques to eliminate the redundant input. This method uses the parsing-based virtual try-on model as the supervision information to train a student network model, which can generate try-on results without extra input. However, due to the huge knowledge gap between the teacher-student networks, direct distillation makes it difficult for the student network to fully simulate the teacher network. To solve this problem, we propose a progressive distillation scheme for image-based virtual try-on called PD-VTON, using an assistant network to alleviate this huge knowledge gap. Our method can generate more realistic and reasonable try-on results without requiring extra body parsing information or body pose information. Specifically, unlike existing distillation-based parsing-free virtual try-on methods, we adopt an assistant-supported progressive distillation network to alleviate the insufficient learning caused by the large knowledge gap and design an Adaptive Choose Teacher (ACT) module to optimize the distillation. Moreover, we introduce a novel cross attention-based stitching structure when generating try-on images, aiming to better constrain the alignment between the high-level semantic features of person image and warped clothing image, and use a transformer-assisted generator to generate the results. Finally, extensive experimental evaluations demonstrate the unique advantages of our method.

InterTeach: A Novel Approach for Semi-Supervised Medical Image Segmentation Using Cooperative Teacher-Student Networks

InterTeach: A Novel Approach for Semi-Supervised Medical Image Segmentation Using Cooperative Teacher-Student Networks

ADPS: Asymmetric Distillation Postsegmentation for Image Anomaly Detection.

Knowledge distillation-based anomaly detection (KDAD) methods rely on the teacher-student paradigm to detect and segment anomalous regions by contrasting the unique features extracted by both networks. However, existing KDAD methods suffer from two main limitations: 1) the student network can effortlessly replicate the teacher network's representations and 2) the features of the teacher network serve solely as a "reference standard" and are not fully leveraged. Toward this end, we depart from the established paradigm and instead propose an innovative approach called asymmetric distillation postsegmentation (ADPS). Our ADPS employs an asymmetric distillation paradigm that takes distinct forms of the same image as the input of the teacher-student networks, driving the student network to learn discriminating representations for anomalous regions. Meanwhile, a customized Weight Mask Block (WMB) is proposed to generate a coarse anomaly localization mask that transfers the distilled knowledge acquired from the asymmetric paradigm to the teacher network. Equipped with WMB, the proposed postsegmentation module (PSM) can effectively detect and segment abnormal regions with fine structures and clear boundaries. Experimental results demonstrate that the proposed ADPS outperforms the state-of-the-art methods in detecting and segmenting anomalies. Surprisingly, ADPS significantly improves average precision (AP) metric by 9% and 20% on the MVTec anomaly detection (AD) and KolektorSDD2 datasets, respectively.

Self-Supervised Sub-Action Parsing Network for Semi-Supervised Action Quality Assessment.

Semi-supervised Action Quality Assessment (AQA) using limited labeled and massive unlabeled samples to achieve high-quality assessment is an attractive but challenging task. The main challenge relies on how to exploit solid and consistent representations of action sequences for building a bridge between labeled and unlabeled samples in the semi-supervised AQA. To address the issue, we propose a Self-supervised sub-Action Parsing Network (SAP-Net) that employs a teacher-student network structure to learn consistent semantic representations between labeled and unlabeled samples for semi-supervised AQA. We perform actor-centric region detection and generate high-quality pseudo-labels in the teacher branch and assists the student branch in learning discriminative action features. We further design a self-supervised sub-action parsing solution to locate and parse fine-grained sub-action sequences. Then, we present the group contrastive learning with pseudo-labels to capture consistent motion-oriented action features in the two branches. We evaluate our proposed SAP-Net on four public datasets: the MTL-AQA, FineDiving, Rhythmic Gymnastics, and FineFS datasets. The experiment results show that our approach outperforms state-of-the-art semi-supervised methods by a significant margin.

Driver distraction detection using semi-supervised lightweight vision transformer

The continuously increasing number of traffic accidents necessitates addressing distracted driving, which is responsible for numerous fatalities. Enhancing driver behavior recognition, particularly through developing a highly reliable Advanced Driver Assistance System (ADAS), holds substantial potential for cultivating safer transportation systems. Inspired by the success of Convolutional Neural Networks (CNNs), various networks have been proposed to enhance the detection accuracy of distracting behaviors. However, existing models have too many parameters and rely heavily on extensive labeled data, leading to time-consuming labeling and large models. To address these limitations, we propose a distracted behavior detection method based on a lightweight vision transformer trained using pseudo-label-based semi-supervised learning to learn discriminative representations from labeled and unlabeled data while ensuring a small model and speedy inference. Specifically, we create strong and weak augmented versions of the input minibatch and employ a hybrid lightweight transformer model in a teacher-student network. Pseudo-labels are generated from the teacher's predictions on weakly augmented data. The student model aligns with these labels on strongly augmented input, with teacher parameters evolving through exponential moving average. Our method presents a real-time, accurate solution for distracted driver detection, with the potential to significantly enhance road safety by reducing accidents. The effectiveness of the proposed approach is demonstrated through a comparative evaluation against alternative fully-supervised and semi-supervised methods. Furthermore, our method is evaluated in naturalistic driving settings with varying lighting and complex backgrounds. Experiments conducted on two publicly available driver distraction detection datasets show that our method outperforms current state-of-the-art approaches.

Joint recognition of basic and compound facial expressions by mining latent soft labels

Previous works on facial expression recognition focus on basic emotions, while ignoring more complex compound expressions. However, both basic and compound emotions appear in the real-world environment. In this work, we aim to jointly recognize basic and compound expressions. Aiming at the Basic-Compound Facial Expression Recognition (BC-FER) task, we illustrate that traditional hard label training is not ideal due to great label dependencies. Therefore, we propose an expression soft label mining (ESLM) method to improve the performance. On the one hand, an iterated soft label mining (ISLM) algorithm assisted by teacher–student network is proposed to make the network generate soft targets automatically for learning. On the other hand, to explicitly leverage prior knowledge of label correlations, we propose an expression correlation score learning (ECSL) loss to regularize the predicted distributions. Extensive experimental results on CFEE, RAF-DB, and EmotioNet show that our method achieves state-of-the-art performance on BC-FER task.

Frame-Level Teacher-Student Learning With Data Privacy for EEG Emotion Recognition.

Recently, electroencephalogram (EEG) emotion recognition has gradually attracted a lot of attention. This brief designs a novel frame-level teacher-student framework with data privacy (FLTSDP) for EEG emotion recognition. The framework first proposes a teacher-student network without prior professional information for automated filtering of useful frame-level features by a gated mechanism and extracting high-level features by using knowledge distillation to capture the results of EEG emotion recognition from a teacher network and student networks. Then, the results from subnetworks are integrated by using the novel decision module, which, motivated by the voting mechanism, adjusts the composition of feature vectors and improves the weight of accurate prediction to optimize the integration effect. During training, an innovative data privacy protection mechanism is applied for avoiding data sharing, where each student network only inherits weights from all trained networks and does not inherit the training dataset. Here, the framework can be repeatedly optimized and improved by only training the next student subnetwork on new EEG signals. Experimental results show that our framework improves the accuracy of EEG emotion recognition by more than 5% and gets state-of-the-art performance for EEG emotion recognition in the subject-independent mode.

Bidirectional Semi-supervised Dual-branch CNN for Robust 3D Reconstruction of Stereo Endoscopic Images via Adaptive Cross and Parallel Supervisions.

Semi-supervised learning via teacher-student network can train a model effectively on a few labeled samples. It enables a student model to distill knowledge from the teacher's predictions of extra unlabeled data. However, such knowledge flow is typically unidirectional, having the accuracy vulnerable to the quality of teacher model. In this paper, we seek to robust 3D reconstruction of stereo endoscopic images by proposing a novel fashion of bidirectional learning between two learners, each of which can play both roles of teacher and student concurrently. Specifically, we introduce two self-supervisions, i.e., Adaptive Cross Supervision (ACS) and Adaptive Parallel Supervision (APS), to learn a dual-branch convolutional neural network. The two branches predict two different disparity probability distributions for the same position, and output their expectations as disparity values. The learned knowledge flows across branches along two directions: a cross direction (disparity guides distribution in ACS) and a parallel direction (disparity guides disparity in APS). Moreover, each branch also learns confidences to dynamically refine its provided supervisions. In ACS, the predicted disparity is softened into a unimodal distribution, and the lower the confidence, the smoother the distribution. In APS, the incorrect predictions are suppressed by lowering the weights of those with low confidence. With the adaptive bidirectional learning, the two branches enjoy well-tuned mutual supervisions, and eventually converge on a consistent and more accurate disparity estimation. The experimental results on four public datasets demonstrate our superior accuracy over other state-of-the-arts with a relative decrease of averaged disparity error by at least 9.76%.

Gradient aligned domain generalization with a mutual teaching teacher-student network for intelligent fault diagnosis

Gradient aligned domain generalization with a mutual teaching teacher-student network for intelligent fault diagnosis

Attention and feature transfer based knowledge distillation

Existing knowledge distillation (KD) methods are mainly based on features, logic, or attention, where features and logic represent the results of reasoning at different stages of a convolutional neural network, and attention maps symbolize the reasoning process. Because of the continuity of the two in time, transferring only one of them to the student network will lead to unsatisfactory results. We study the knowledge transfer between the teacher-student network to different degrees, revealing the importance of simultaneously transferring knowledge related to the reasoning process and reasoning results to the student network, providing a new perspective for the study of KD. On this basis, we proposed the knowledge distillation method based on attention and feature transfer (AFT-KD). First, we use transformation structures to transform intermediate features into attentional and feature block (AFB) that contain both inference process information and inference outcome information, and force students to learn the knowledge in AFBs. To save computation in the learning process, we use block operations to align the teacher-student network. In addition, in order to balance the attenuation ratio between different losses, we design an adaptive loss function based on the loss optimization rate. Experiments have shown that AFT-KD achieves state-of-the-art performance in multiple benchmark tests.