Teacher Network Research Articles

Learn How to See: Collaborative Embodied Learning for Object Detection and Camera Adjusting

Passive object detectors, trained on large-scale static datasets, often overlook the feedback from object detection to image acquisition. Embodied vision and active detection mitigate this issue by interacting with the environment. Nevertheless, the materialization of activeness hinges on resource-intensive data collection and annotation. To tackle these challenges, we propose a collaborative student-teacher framework. Technically, a replay buffer is built based on the trajectory data to encapsulate the relationship of state, action, and reward. In addition, the student network diverges from reinforcement learning by redefining sequential decision pathways using a GPT structure enriched with causal self-attention. Moreover, the teacher network establishes a subtle state-reward mapping based on adjacent benefit differences, providing reliable rewards for student adaptively self-tuning with the vast unlabeled replay buffer data. Additionally, an innovative yet straightforward benefit reference value is proposed within the teacher network, adding to its effectiveness and simplicity. Leveraging a flexible replay buffer and embodied collaboration between teacher and student, the framework learns to see before detection with shallower features and shorter inference steps. Experiments highlight significant advantages of our algorithm over state-of-the-art detectors. The code is released at https://github.com/lydonShen/STF.

Discrepancy and Uncertainty Aware Denoising Knowledge Distillation for Zero-Shot Cross-Lingual Named Entity Recognition

The knowledge distillation-based approaches have recently yielded state-of-the-art (SOTA) results for cross-lingual NER tasks in zero-shot scenarios. These approaches typically employ a teacher network trained with the labelled source (rich-resource) language to infer pseudo-soft labels for the unlabelled target (zero-shot) language, and force a student network to approximate these pseudo labels to achieve knowledge transfer. However, previous works have rarely discussed the issue of pseudo-label noise caused by the source-target language gap, which can mislead the training of the student network and result in negative knowledge transfer. This paper proposes an discrepancy and uncertainty aware Denoising Knowledge Distillation model (DenKD) to tackle this issue. Specifically, DenKD uses a discrepancy-aware denoising representation learning method to optimize the class representations of the target language produced by the teacher network, thus enhancing the quality of pseudo labels and reducing noisy predictions. Further, DenKD employs an uncertainty-aware denoising method to quantify the pseudo-label noise and adjust the focus of the student network on different samples during knowledge distillation, thereby mitigating the noise's adverse effects. We conduct extensive experiments on 28 languages including 4 languages not covered by the pre-trained models, and the results demonstrate the effectiveness of our DenKD.

Asymmetric Mutual Alignment for Unsupervised Zero-Shot Sketch-Based Image Retrieval

In recent years, many methods have been proposed to address the zero-shot sketch-based image retrieval (ZS-SBIR) task, which is a practical problem in many applications. However, in real-world scenarios, on the one hand, we can not obtain training data with the same distribution as the test data, and on the other hand, the labels of training data are not available as usual. To tackle this issue, we focus on a new problem, namely unsupervised zero-shot sketch-based image retrieval (UZS-SBIR), where the available training data does not have labels while the training and testing categories are not overlapping. In this paper, we introduce a new asymmetric mutual alignment method (AMA) including a self-distillation module and a cross-modality mutual alignment module. First, we conduct self-distillation to extract the feature embeddings from unlabeled data. Due to the lack of available information in an unsupervised manner, we employ the cross-modality mutual alignment module to further excavate underlying intra-modality and inter-modality relationships from unlabeled data, and take full advantage of these correlations to align the feature embeddings in image and sketch domains. Meanwhile, the feature representations are enhanced by the intra-modality clustering relations, leading to better generalization ability to unseen classes. Moreover, we conduct an asymmetric strategy to update the teacher and student networks, respectively. Extensive experimental results on several benchmark datasets demonstrate the superiority of our method.

DistilVPR: Cross-Modal Knowledge Distillation for Visual Place Recognition

The utilization of multi-modal sensor data in visual place recognition (VPR) has demonstrated enhanced performance compared to single-modal counterparts. Nonetheless, integrating additional sensors comes with elevated costs and may not be feasible for systems that demand lightweight operation, thereby impacting the practical deployment of VPR. To address this issue, we resort to knowledge distillation, which empowers single-modal students to learn from cross-modal teachers without introducing additional sensors during inference. Despite the notable advancements achieved by current distillation approaches, the exploration of feature relationships remains an under-explored area. In order to tackle the challenge of cross-modal distillation in VPR, we present DistilVPR, a novel distillation pipeline for VPR. We propose leveraging feature relationships from multiple agents, including self-agents and cross-agents for teacher and student neural networks. Furthermore, we integrate various manifolds, characterized by different space curvatures for exploring feature relationships. This approach enhances the diversity of feature relationships, including Euclidean, spherical, and hyperbolic relationship modules, thereby enhancing the overall representational capacity. The experiments demonstrate that our proposed pipeline achieves state-of-the-art performance compared to other distillation baselines. We also conduct necessary ablation studies to show design effectiveness. The code is released at: https://github.com/sijieaaa/DistilVPR

CSL: Class-Agnostic Structure-Constrained Learning for Segmentation Including the Unseen

Addressing Out-Of-Distribution (OOD) Segmentation and Zero-Shot Semantic Segmentation (ZS3) is challenging, necessitating segmenting unseen classes. Existing strategies adapt the class-agnostic Mask2Former (CA-M2F) tailored to specific tasks. However, these methods cater to singular tasks, demand training from scratch, and we demonstrate certain deficiencies in CA-M2F, which affect performance. We propose the Class-Agnostic Structure-Constrained Learning (CSL), a plug-in framework that can integrate with existing methods, thereby embedding structural constraints and achieving performance gain, including the unseen, specifically OOD, ZS3, and domain adaptation (DA) tasks. There are two schemes for CSL to integrate with existing methods (1) by distilling knowledge from a base teacher network, enforcing constraints across training and inference phrases, or (2) by leveraging established models to obtain per-pixel distributions without retraining, appending constraints during the inference phase. Our soft assignment and mask split methodologies enhance OOD object segmentation. Empirical evaluations demonstrate CSL's prowess in boosting the performance of existing algorithms spanning OOD segmentation, ZS3, and DA segmentation, consistently transcending the state-of-art across all three tasks.

EPSD: Early Pruning with Self-Distillation for Efficient Model Compression

Neural network compression techniques, such as knowledge distillation (KD) and network pruning, have received increasing attention. Recent work `Prune, then Distill' reveals that a pruned student-friendly teacher network can benefit the performance of KD. However, the conventional teacher-student pipeline, which entails cumbersome pre-training of the teacher and complicated compression steps, makes pruning with KD less efficient. In addition to compressing models, recent compression techniques also emphasize the aspect of efficiency. Early pruning demands significantly less computational cost in comparison to the conventional pruning methods as it does not require a large pre-trained model. Likewise, a special case of KD, known as self-distillation (SD), is more efficient since it requires no pre-training or student-teacher pair selection. This inspires us to collaborate early pruning with SD for efficient model compression. In this work, we propose the framework named Early Pruning with Self-Distillation (EPSD), which identifies and preserves distillable weights in early pruning for a given SD task. EPSD efficiently combines early pruning and self-distillation in a two-step process, maintaining the pruned network's trainability for compression. Instead of a simple combination of pruning and SD, EPSD enables the pruned network to favor SD by keeping more distillable weights before training to ensure better distillation of the pruned network. We demonstrated that EPSD improves the training of pruned networks, supported by visual and quantitative analyses. Our evaluation covered diverse benchmarks (CIFAR-10/100, Tiny-ImageNet, full ImageNet, CUB-200-2011, and Pascal VOC), with EPSD outperforming advanced pruning and SD techniques.

FD3D: Exploiting Foreground Depth Map for Feature-Supervised Monocular 3D Object Detection

Monocular 3D object detection usually adopts direct or hierarchical label supervision. Recently, the distillation supervision transfers the spatial knowledge from LiDAR- or stereo-based teacher networks to monocular detectors, but remaining the domain gap. To mitigate this issue and pursue adequate label manipulation, we exploit Foreground Depth map for feature-supervised monocular 3D object detection named FD3D, which develops the high-quality instructive intermediate features to conduct desirable auxiliary feature supervision with only the original image and annotation foreground object-wise depth map (AFOD) as input. Furthermore, we build up our instructive feature generation network to create instructive spatial features based on the sufficient correlation between image features and pre-processed AFOD, where AFOD provides the attention focus only on foreground objects to achieve clearer guidance in the detection task. Moreover, we apply the auxiliary feature supervision from the pixel and distribution level to achieve comprehensive spatial knowledge guidance. Extensive experiments demonstrate that our method achieves state-of-the-art performance on both the KITTI and nuScenes datasets, with no external data and no extra inference computational cost. We also conduct quantitative and qualitative studies to reveal the effectiveness of our designs.

How a student becomes a teacher: learning and forgetting through spectral methods

In theoretical machine learning, the teacher–student paradigm is often employed as an effective metaphor for real-life tuition. A student network is trained on data generated by a fixed teacher network until it matches the instructor’s ability to cope with the assigned task. The above scheme proves particularly relevant when the student network is overparameterized (namely, when larger layer sizes are employed) as compared to the underlying teacher network. Under these operating conditions, it is tempting to speculate that the student ability to handle the given task could be eventually stored in a sub-portion of the whole network. This latter should be to some extent reminiscent of the frozen teacher structure, according to suitable metrics, while being approximately invariant across different architectures of the student candidate network. Unfortunately, state-of-the-art conventional learning techniques could not help in identifying the existence of such an invariant subnetwork, due to the inherent degree of non-convexity that characterizes the examined problem. In this work, we take a decisive leap forward by proposing a radically different optimization scheme which builds on a spectral representation of the linear transfer of information between layers. The gradient is hence calculated with respect to both eigenvalues and eigenvectors with negligible increase in terms of computational and complexity load, as compared to standard training algorithms. Working in this framework, we could isolate a stable student substructure, that mirrors the true complexity of the teacher in terms of computing neurons, path distribution and topological attributes. When pruning unimportant nodes of the trained student, as follows a ranking that reflects the optimized eigenvalues, no degradation in the recorded performance is seen above a threshold that corresponds to the effective teacher size. The observed behavior can be pictured as a genuine second-order phase transition that bears universality traits. Code is available at: https://github.com/Jamba15/Spectral-regularization-teacher-student/tree/master.

Semi-supervised learning approach for construction object detection by integrating super-resolution and mean teacher network

Deep learning-based object detection methods are utilized for safety management at construction sites, which require large-scale, high-quality, and well-labeled datasets for training. The existing construction datasets are relatively small due to the high expense of labor-intensive annotation, and the varying quality of the construction images also affects the detection performance of the model. To address the limitations of datasets, this study proposes a new method for construction object detection by integrating super-resolution and semi-supervised learning. The proposed method improves the quality of construction images and achieves excellent detection performance with limited labeled data. First, the Real-ESRGAN model is introduced to improve the quality of construction images and make the construction objects visible. The proposed super-resolution method can enhance the texture details of low-resolution images, hence improving the performance of object detection models. Second, the mean-teacher network is adopted to expand the training set, thus avoiding the labor-intensive annotation work. To verify the effectiveness of the proposed method, the method is applied to the state-of-the-art Yolov5 object detection model, and construction images from the Site Object Detection Dataset (SODA) with different labeled data proportions (from 10% to 50% in 10% intervals with an extreme case of 5%) are used as the training set. By comparing with the existing supervised learning method, it is shown that the proposed method can achieve better detection performance. In particular, the method is more effective in enhancing detection performance when the proportion of the labeled data is smaller, which is of great practical value in real-world engineering. The experimental results show the potential of the proposed method in improving image quality and reducing the expense of developing construction datasets.

Contrastive learning-based knowledge distillation for RGB-thermal urban scene semantic segmentation

RGB thermal semantic segmentation facilitates unmanned platforms to perceive and characterize their surrounding environment, which is critical for autonomous driving tasks. Deep-learning-based algorithms have achieved dominance in terms of accuracy and robustness. However, their large parameter sizes and significant computational demands impede their application in terminal devices. To address this challenge, we propose a novel strategy for achieving a balance between effectiveness and compactness. It includes a robust teacher network, CLNet-T, and a streamlined student network, CLNet-S. Using knowledge distillation (KD), we obtained an optimized model called CLNet-S*. Specifically, CLNet-T and CLNet-S were identical in all aspects except for the feature extraction component. They included a multi-attribute hierarchical feature interaction module (MHFI) and a detail-guided semantic decoder (DGSD). The MHFI initially filters features by considering the characteristics of the low- and high-level features. It gradually combines complementary and common features from various modalities in distinct receptive fields. DGSD uses edge and distribution information to guide semantic decoding, thereby improving the segmentation accuracy at class boundaries. To enhance the performance of the compact student model, our KD strategy includes detail, semantic response distillation (DSRD), and contrastive learning-based feature distillation (CLFD). Practically, DSRD enables the student model to gain knowledge from the teacher model at both the detailed and semantic levels. At the same time, CLFD increases the similarity of features within the same categories and emphasizes the distinctiveness of features between different categories in both the student and teacher models. Extensive experiments conducted on two standard datasets have consistently demonstrated that both CLNet-T and CLNet-S* outperform other state-of-the-art methods. The code and results are available at https://github.com/xiaodonguo/CLNet.

Masked feature regeneration based asymmetric student–teacher network for anomaly detection

Masked feature regeneration based asymmetric student–teacher network for anomaly detection

Collaborative Filter Pruning for Efficient Automatic Surface Defect Detection

Surface defect detection is a critical task in industrial production, and numerous methods have been proposed to achieve high detection accuracy. Although deep learning-based approaches have achieved state-of-the-art performances, their vast computational cost and high memory footprint prevent their deployment in resource-constrained environments. To address this problem, we propose a collaborative filter pruning method for the defect detection model, which significantly reduces the number of required calculations and parameters while maintaining high performance, even in cases with tasks suffering from the class imbalance problem. Our method aims to obtain lightweight pruned models by removing unimportant filters according to their importance evaluated by both structural similarity and detail richness of corresponding feature maps. Moreover, to improve the performance of pruned models, we propose a knowledge-fused fine-tuning approach that fuses the knowledge derived from two teacher networks to look after both representation learning and classifier learning, alleviating the class imbalance problem. Experimental results on four public datasets demonstrate that the proposed approach performs favorably relative to the state-of-the-art methods. In particular, the proposed method achieves 39× and 59× parameter compression for VGG-16 and ResNet-50, respectively, on the NEU-CLS dataset, with a very small detection accuracy loss (< 0.2%). The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/HXuan-Wang/CFP</uri> .

Blended and online learning: what works and why in primary and secondary schools. Evidence from Northern Ireland

ABSTRACT Data from a sample of teachers, parents and pupils in Northern Ireland about their experiences of blended and online learning during the pandemic has thrown light on effective practice and some of the reasons for this. After an initial period of uncertainty and confusion, most of the five schools studied found ways to provide regular online lessons which in many cases had high levels of interaction between teachers and pupils and in some cases between pupils. This was true also in schools which had high levels of pupil disadvantage measured through entitlement to free school meals. The reasons for the emergence of good practice were the prior investment in ICT facilities for every school, the support provided through educational authorities and local teacher networks, and the quality of leadership in schools.

Latent domain knowledge distillation for nighttime semantic segmentation

Despite significant progress made in image semantic segmentation, most research has primarily focused on daytime scenes. Semantic segmentation of nighttime images is equally critical for autonomous driving; however, this task poses more challenges due to inadequate lighting and difficulties associated with obtaining accurate manual annotations. In this paper, we propose a novel method called Latent Domain Knowledge Distillation (LDKD) for accurate nighttime semantic segmentation. Within a teacher-student framework, our LDKD combines domain alignment and knowledge distillation in a mutually reinforcing manner. First, we introduce a bidirectional photometric alignment module capable of generating daytime-like and nighttime-like latent images in real-time. This module effectively bridges the appearance gap between the source domain (daytime) and the target domain (nighttime). Second, we enhance collaboration between the teacher and student networks. The student network unifies image-level and feature-level domain alignment, while the teacher network generates reliable pseudo-labels by distilling knowledge from the latent domain. Furthermore, to account for potential noise in pseudo-labels, we propose a noise-tolerant learning method aimed at mitigating the risks associated with overreliance on pseudo-labels during model training. Extensive experiments on Dark Zurich and ACDC datasets show that our LDKD achieves state-of-the-art performance, demonstrating the effectiveness of our method for nighttime semantic segmentation.

Effective Approaches to Teacher Professional Development in Thailand: Case Study

This research aimed to investigate effective approaches to teacher development as a meaningful form of professional development (PD). The objective was to assess the impact of teachers&amp;rsquo; learning as an approach to PD and to summarize best practices. The study involved teachers from two primary schools who participated in the Healthy School Empowerment Project, a four-year collaborative learning initiative designed to support educators in improving classroom instruction and enhancing school performance. The findings revealed the most effective approaches for professional development and enhancing teachers&amp;rsquo; learning include establishing the school as a professional learning community (PLC), implementing teacher-as-learner programs, providing coaching by master teachers, and fostering teacher networks. Bottom of Form

A deep learning knowledge distillation framework using knee MRI and arthroscopy data for meniscus tear detection.

Purpose: To construct a deep learning knowledge distillation framework exploring the utilization of MRI alone or combing with distilled Arthroscopy information for meniscus tear detection. Methods: A database of 199 paired knee Arthroscopy-MRI exams was used to develop a multimodal teacher network and an MRI-based student network, which used residual neural networks architectures. A knowledge distillation framework comprising the multimodal teacher network T and the monomodal student network S was proposed. We optimized the loss functions of mean squared error (MSE) and cross-entropy (CE) to enable the student network S to learn arthroscopic information from the teacher network T through our deep learning knowledge distillation framework, ultimately resulting in a distilled student network S T. A coronal proton density (PD)-weighted fat-suppressed MRI sequence was used in this study. Fivefold cross-validation was employed, and the accuracy, sensitivity, specificity, F1-score, receiver operating characteristic (ROC) curves and area under the receiver operating characteristic curve (AUC) were used to evaluate the medial and lateral meniscal tears detection performance of the models, including the undistilled student model S, the distilled student model S T and the teacher model T. Results: The AUCs of the undistilled student model S, the distilled student model S T, the teacher model T for medial meniscus (MM) tear detection and lateral meniscus (LM) tear detection are 0.773/0.672, 0.792/0.751 and 0.834/0.746, respectively. The distilled student model S T had higher AUCs than the undistilled model S. After undergoing knowledge distillation processing, the distilled student model demonstrated promising results, with accuracy (0.764/0.734), sensitivity (0.838/0.661), and F1-score (0.680/0.754) for both medial and lateral tear detection better than the undistilled one with accuracy (0.734/0.648), sensitivity (0.733/0.607), and F1-score (0.620/0.673). Conclusion: Through the knowledge distillation framework, the student model S based on MRI benefited from the multimodal teacher model T and achieved an improved meniscus tear detection performance.

A progressive distillation network for practical image-based virtual try-on

The image-based virtual try-on technology aims to match in-store clothing to person's image wearing clothes. Previous methods require a large amount of input information for each try-on result, which is not practical, and slight deviations in input information can cause a large number of artifacts in the try-on results. In recent years, researchers have paid attention to parsing-free virtual try-on methods, and a groundbreaking work used knowledge distillation techniques to eliminate the redundant input. This method uses the parsing-based virtual try-on model as the supervision information to train a student network model, which can generate try-on results without extra input. However, due to the huge knowledge gap between the teacher-student networks, direct distillation makes it difficult for the student network to fully simulate the teacher network. To solve this problem, we propose a progressive distillation scheme for image-based virtual try-on called PD-VTON, using an assistant network to alleviate this huge knowledge gap. Our method can generate more realistic and reasonable try-on results without requiring extra body parsing information or body pose information. Specifically, unlike existing distillation-based parsing-free virtual try-on methods, we adopt an assistant-supported progressive distillation network to alleviate the insufficient learning caused by the large knowledge gap and design an Adaptive Choose Teacher (ACT) module to optimize the distillation. Moreover, we introduce a novel cross attention-based stitching structure when generating try-on images, aiming to better constrain the alignment between the high-level semantic features of person image and warped clothing image, and use a transformer-assisted generator to generate the results. Finally, extensive experimental evaluations demonstrate the unique advantages of our method.

MTUW-GAN: A Multi-Teacher Knowledge Distillation Generative Adversarial Network for Underwater Image Enhancement

Underwater imagery is plagued by issues such as image blurring and color distortion, which significantly impede the detection and operational capabilities of underwater robots, specifically Autonomous Underwater Vehicles (AUVs). Previous approaches to image fusion or multi-scale feature fusion based on deep learning necessitated multi-branch image preprocessing prior to merging through fusion modules. However, these methods have intricate network structures and a high demand for computational resources, rendering them unsuitable for deployment on AUVs, which have limited resources at their disposal. To tackle these challenges, we propose a multi-teacher knowledge distillation GAN for underwater image enhancement (MTUW-GAN). Our approach entails multiple teacher networks instructing student networks simultaneously, enabling them to enhance color and detail in degraded images from various perspectives, thus achieving an image-fusion-level performance. Additionally, we employ middle layer channel distillation in conjunction with the attention mechanism to extract and transfer rich middle layer feature information from the teacher model to the student model. By eliminating multiplexed branching and fusion modules, our lightweight student model can directly generate enhanced underwater images through model compression. Furthermore, we introduce a multimodal objective enhancement function to refine the overall framework training, striking a balance between a low computational effort and high-quality image enhancement. Experimental results, obtained by comparing our method with existing approaches, demonstrate the clear advantages of our proposed method in terms of visual quality, model parameters, and real-time performance. Consequently, our method serves as an effective solution for real-time underwater image enhancement, specifically tailored for deployment on AUVs.

Pan-sharpening via intrinsic decomposition knowledge distillation

Existing deep-learning-based pan-sharpening strategies mainly involve the fusion of panchromatic and multispectral (MS) information at both the pixel and feature levels. In this paper, we hypothesize that the MS image can be expressed as the multiplication of reflectance and illumination components, and that the reflection components of low-resolution (LR) MS and high-resolution (HR) MS images are invariant. Here, the spectral reflection component can effectively describe the spectral response of an object, while the illumination component can effectively describe its texture. Based on this hypothesis, we propose a novel and concise pan-sharpening framework called intrinsic decomposition knowledge distillation. Specifically, the teacher network decomposes the HR MS image into reflectance and illumination components, which are then combined in the student network with the reflectance component and the enhanced illumination component from LR MS to reconstruct the pan-sharpened image. To approximate the component distributions from the teacher network, we introduce a novel three-stage knowledge distillation strategy that can transfer knowledge about the relationships between components and constrain the student network. Our quantitative and qualitative comparisons demonstrate the reasonableness of our hypothesis and the effectiveness of our proposed method in significantly improving perception quality.

Distilling object detectors with efficient logit mimicking and mask-guided feature imitation

Knowledge distillation (KD) is a promising approach to learning compact models for object detection with information inherited from intricate teacher networks. In this paper, we raise some shortcomings of existing KD methods for object detectors, e.g., ignoring knowledge selection, coarse feature imitation mask, etc. To address these issues, a novel KD framework has been presented to train efficient object detectors via Logit Mimicking and Feature Imitation (LMFI). First, a novel logit mimicking method is put forward to distill classification and localization heads. On the one hand, it first proposes to mimic the classification logits of one-category object detectors. On the other hand, the localization knowledge from teacher predictions and ground truths are exploited, which dynamically guides the learning of student’s regression outputs by stages. Second, an adaptive positive teacher selection (APTS) strategy is designed to obtain high-quality teacher samples during distillation, which reduces the transmission of inferior knowledge. Moreover, a soft metric and a fine-grained mask are heuristically introduced to reconcile the discrepancy between teacher and student features in a position-wise manner. Extensive experiments show that LMFI outperforms the state-of-the-art KD frameworks for object detection. It can significantly boost the performance of various detectors on different benchmarks, e.g., 2.83% and 2.58% MR−2 reduction of Cascade R-CNN on the S and All sets of CityPersons, and an improvement of ResNet-50 based TOOD from 40.3% to 42.9% mAP on the COCO benchmark.