Teacher Network Research Articles

PURSUhInT: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation

One of the most efficient methods for model compression is hint distillation, where the student model is injected with information (hints) from several different layers of the teacher model. Although the selection of hint points can drastically alter the compression performance, conventional distillation approaches overlook this fact and use the same hint points as in the early studies. Therefore, we propose a clustering based hint selection methodology, where the layers of teacher model are clustered with respect to several metrics and the cluster centers are used as the hint points. Our method is applicable for any student network, once it is applied on a chosen teacher network. The proposed approach is validated in CIFAR-100 and ImageNet datasets, using various teacher–student pairs and numerous hint distillation methods. Our results show that hint points selected by our algorithm results in superior compression performance compared to state-of-the-art knowledge distillation algorithms on the same student models and datasets.

Assessing the learning of pupils with special educational needs and disabilities in mainstream school physical education

AbstractIn the UK, one consequence of neoliberalism has been the development of test cultures in schools and standardised assessment strategies used to judge all pupils against within and across curriculum subjects. Few studies to date have explored the influence of this on assessing the learning of pupils with special educational needs and disabilities (SEND), and none have centred physical education (PE). This study used the concept of ableism and semi‐structured interviews to explore mainstream secondary school PE teachers’ views and experiences of assessing the learning of pupils with SEND. Based on the findings, we discuss the importance of schools disrupting hegemonic, ableist modes of thinking that cast pupils with SEND as being of inferior ability when compared with their peers and thus being disadvantaged by standardised, normative assessment practices. Specifically, we identify a need for senior leaders and teachers in schools to recognise the needs and capabilities of pupils with SEND, through more holistic assessment approaches that focus on social, affective, cognitive and physical learning and development. We end by discussing the significance of initial teacher education and teacher networks to support this endeavour and advocating for the amplification of the voices of pupils with SEND, given that they have expert knowledge about the perceived inclusivity of assessment in PE because they can draw upon their lived and embodied experiences.

Knowledge distillation technique enabled hardware efficient OSNR monitoring from directly detected PDM-QAM signals

Deep neural network (DNN) enabled optical SNR (OSNR) monitoring is vital to facilitate the management of agile optical networks. However, both the scale size and the implementation complexity of DNNs prevent their practical application. Here, we experimentally demonstrate the knowledge distillation technique (KDT) for DNN-based OSNR monitoring from the directly detected polarization division multiplexed 16 quadrature amplitude modulation (PDM-16QAM) and PDM-64QAM signals by the use of amplitude histograms. First, both floating-point operations (FLOPs) and parameters (Params) are introduced into the fiber optic communications as the evaluation metrics for the computational complexity of DNNs. Then, the pre-trained teacher network and the to-be-used student network constitute a KDT network, while the task of the pre-trained teacher network transfers the knowledge to the student network. From the experimental results, we learn that the KDT benefits the student network and realizes a rms error (RMSE) of 0.98 dB over OSNR ranges of 14–24 dB and 23–34 dB for 10 Gbaud PDM-16QAM and PDM-64QAM signals, respectively. The Params of the student network are only 0.004M, and the FLOPs are 0.25M, which are less than that of existing OSNR monitoring schemes. Meanwhile, an RMSE improvement of 0.23 dB arising in the OSNR monitoring has been secured to the student network with the help of KDT. Therefore, we believe our proposed KDT is promising for optical performance monitoring with a lightweight DNN.

A Class-Incremental Detection Method of Remote Sensing Images Based on Selective Distillation

With the rapid development of remote sensing technology and the growing demand for applications, the classical deep learning-based object detection model is bottlenecked in processing incremental data, especially in the increasing classes of detected objects. It requires models to sequentially learn new classes of objects based on the current model, while preserving old categories-related knowledge. Existing class-incremental detection methods achieve this goal mainly by constraining the optimization trajectory in the feature of output space. However, these works neglect the case where the previously learned background is a new category to learn, resulting in performance degradation in the new category because of the conflict between remaining the background-related knowledge or updating the background-related knowledge. This paper proposes a novel class-incremental detection method incorporated with the teacher-student architecture and the selective distillation (SDCID) strategy. Specifically, it is the asymmetry architecture, i.e., the teacher network temporarily stores historical knowledge of previously learned objects, and the student network integrates historical knowledge from the teacher network with the newly learned object-related knowledge, respectively. This asymmetry architecture reveals the significance of the distinct representation of history knowledge and new knowledge in incremental detection. Furthermore, SDCID selectively masks the shared subobject of new images to learn and previously learned background, while learning new categories of images and then transfers the classification results of the student model to the background class following the judgment model of the teacher model. This manner avoids interferences between the background category-related knowledge from a teacher model and the learning of other new classes of objects. In addition, we proposed a new incremental learning evaluation metric, C-SP, to comprehensively evaluate the incremental learning stability and plasticity performance. We verified the proposed method on two object detection datasets of remote sensing images, i.e., DIOR and DOTA. The experience results in accuracy and C-SP suggest that the proposed method surpasses existing class-incremental detection methods. We further analyzed the influence of the mask component in our method and the hyper-parameters sensitive to our method.

Knowledge Distillation Anomaly Detection with Multi-Scale Feature Fusion

To enhance the generalization of anomaly detection, this paper proposes a multi-scale detection method based on knowledge distillation. During training, the well-pretrained teacher network is used to teach the student network to learn the feature of normal samples. During testing, the teacher network can still represent anomaly well due to its strong generalization, while the student network cannot. The difference between them makes the detection task available. Furthermore, a mid-level feature pyramid structure is adopted to enhance the ability for handling the anomaly with different size, and a feature reconstruction modular is also employed to enlarge the difference between teacher and student network for an anomaly. The method achieves 97.8% and 97.7% AUC score on pixel and image level respectively, evaluated on the public benchmark-MVTecAD.

NABT

NABT

Deep Adaptively-Enhanced Hashing With Discriminative Similarity Guidance for Unsupervised Cross-Modal Retrieval

Cross-modal hashing that leverages hash functions to project high-dimensional data from different modalities into the compact common hamming space, has shown immeasurable potential in cross-modal retrieval. To ease labor costs, unsupervised cross-modal hashing methods are proposed. However, existing unsupervised methods still suffer from two factors in the optimization of hash functions: 1) similarity guidance, they barely give a clear definition of whether is similar or not between data points, leading to the residual of the redundant information; 2) optimization strategy, they ignore the fact that the similarity learning abilities of different hash functions are different, which makes the hash function of one modality weaker than the hash function of the other modality. To alleviate such limitations, this paper proposes an unsupervised cross-modal hashing method to train hash functions with discriminative similarity guidance and adaptively-enhanced optimization strategy, termed Deep Adaptively-Enhanced Hashing (DAEH). Specifically, to estimate the similarity relations with discriminability, Information Mixed Similarity Estimation (IMSE) is designed by integrating information from distance distributions and the similarity ratio. Moreover, Adaptive Teacher Guided Enhancement (ATGE) optimization strategy is also designed, which employs information theory to discover the weaker hash function and utilizes an extra teacher network to enhance it. Extensive experiments on three benchmark datasets demonstrate the superiority of the proposed DAEH against the state-of-the-arts.

D2-Net: Dual Disentanglement Network for Brain Tumor Segmentation With Missing Modalities

Multi-modal Magnetic Resonance Imaging (MRI) can provide complementary information for automatic brain tumor segmentation, which is crucial for diagnosis and prognosis. While missing modality data is common in clinical practice and it can result in the collapse of most previous methods relying on complete modality data. Current state-of-the-art approaches cope with the situations of missing modalities by fusing multi-modal images and features to learn shared representations of tumor regions, which often ignore explicitly capturing the correlations among modalities and tumor regions. Inspired by the fact that modality information plays distinct roles to segment different tumor regions, we aim to explicitly exploit the correlations among various modality-specific information and tumor-specific knowledge for segmentation. To this end, we propose a Dual Disentanglement Network (D2-Net) for brain tumor segmentation with missing modalities, which consists of a modality disentanglement stage (MD-Stage) and a tumor-region disentanglement stage (TD-Stage). In the MD-Stage, a spatial-frequency joint modality contrastive learning scheme is designed to directly decouple the modality-specific information from MRI data. To decompose tumor-specific representations and extract discriminative holistic features, we propose an affinity-guided dense tumor-region knowledge distillation mechanism in the TD-Stage through aligning the features of a disentangled binary teacher network with a holistic student network. By explicitly discovering relations among modalities and tumor regions, our model can learn sufficient information for segmentation even if some modalities are missing. Extensive experiments on the public BraTS-2018 database demonstrate the superiority of our framework over state-of-the-art methods in missing modalities situations. Codes are available at https://github.com/CityU-AIM-Group/D2Net.

Improved Knowledge Distillationwith Dynamic Network Pruning

Deploying convolutional neural networks to mobile or embedded devices is often prohibited by limited memory and computational resources. This is particularly problematic for the most successful networks, which tend to be very large and require long inference times. Many alternative approaches have been developed for compressing neural networks based on pruning, regularization, quantization or distillation. In this paper, we propose the “Knowledge Distillation with Dynamic Pruning” (KDDP), which trains a dynamically pruned compact student network under the guidance of a large teacher network. In KDDP, we train the student network with supervision from the teacher network, while applying L1 regularization on the neuron activations in a fully-connected layer. Subsequently, we prune inactive neurons. Our method automatically determines the final size of the student model. We evaluate the compression rate and accuracy of the resulting networks on an image classification dataset, and compare them to results obtained by Knowledge Distillation (KD). Compared to KD, our method produces better accuracy and more compact models.

유아교육현장의 문화예술교육 운영실태 분석

The purpose of this study is to understand of cultural arts education in the early childhood education field through big data analysis. As a result of the study, ‘education’, ‘childcare’, ‘parent’, ‘guide’, and ‘support’ etc were found as key keywords. They were related to the subjects of cultural arts education, program contents and methods, notices and applications. Second, as a result of macroscopic analysis, it was found that the density between keywords was high and the centrality was low. As a result of the microscopic analysis, it was found that connection centrality and prestige centrality were higher than the frequency for infants and young children. Third, the ego network for education, infants, parents, and teachers was analyzed. Dicussion and implications were provided based on the above results.

Relay knowledge distillation for efficiently boosting the performance of shallow networks

To reduce the computational consumption and memory footprint of powerful deep neural networks for applications on edge devices, many model compression methods have been proposed. Among them, knowledge distillation, a topic that has been widely studied, aims at training a lightweight and powerful student network under the guidance of a well-trained but cumbersome teacher network. Most existing knowledge distillation methods focus on how to better define knowledge and the student is forced to mimic the representation space of the teacher, ignoring leveraging the powerful teacher architecture to improve the student intermediate representations for approaching the proficient performance of the teacher. Since the teacher naturally has more powerful feature extraction ability and is always available at the training process of the student, we introduce Relay Knowledge Distillation (ReKD) for efficiently boosting the student performance. Concretely, ReKD transfers the student intermediate representations to the teacher, making the student trained with the teacher in an interactive manner. By directly utilizing the teacher powerful feature extraction ability to improve the student intermediate representations, the knowledge of the teacher is implicitly distilled to the student. Extensive experiments on diverse image classification datasets demonstrate the proposed ReKD can significantly improve the student performance, even making it possible for the student to outperform the teacher.

Lightweight Depth Completion Network with Local Similarity-Preserving Knowledge Distillation.

Depth perception capability is one of the essential requirements for various autonomous driving platforms. However, accurate depth estimation in a real-world setting is still a challenging problem due to high computational costs. In this paper, we propose a lightweight depth completion network for depth perception in real-world environments. To effectively transfer a teacher’s knowledge, useful for the depth completion, we introduce local similarity-preserving knowledge distillation (LSPKD), which allows similarities between local neighbors to be transferred during the distillation. With our LSPKD, a lightweight student network is precisely guided by a heavy teacher network, regardless of the density of the ground-truth data. Experimental results demonstrate that our method is effective to reduce computational costs during both training and inference stages while achieving superior performance over other lightweight networks.

Entrepreneurship and international education: the case of business week project 2018

Entrepreneurship plays an increase role in economic development. Entrepreneurship contributes to the development of innovations, increases economic efficiency, and creates new jobs. Entrepreneurship education and international education seems to be an important antecedent for entrepreneurship intentions. This paper presents an International Intensive Program on Entrepreneurship and confirms the importance of this program concerning the skills, attitudes and competencies acquired or developed by students. Teachers also demonstrated interest and satisfaction in mixed approaches, combining the contribution of teacher knowledge and network, as well as the experience of non-academic entrepreneurs and experts.

Automatic Modulation Classification with Neural Networks via Knowledge Distillation

Deep learning is used for automatic modulation recognition in neural networks, and because of the need for high classification accuracy, deeper and deeper networks are used. However, these are computationally very expensive for neural network training and inference, so its utility in the case of a mobile with memory limitations or weak computational power is questionable. As a result, a trade-off between network depth and network classification accuracy must be considered. To address this issue, we used a knowledge distillation method in this study to improve the classification accuracy of a small network model. First, we trained Inception–Resnet as a teacher network, which has a size of 311.77 MB and a final peak classification accuracy of 93.09%. We used the method to train convolutional neural network 3 (CNN3) and increase its peak classification accuracy from 79.81 to 89.36%, with a network size of 0.37 MB. It was also used similarly to train mini Inception–Resnet and increase its peak accuracy from 84.18 to 93.59%, with a network size of 39.69 MB. When we compared all classification accuracy peaks, we discover that knowledge distillation improved small networks and that the student network had the potential to outperform the teacher network. Using knowledge distillation, a small network model can achieve the classification accuracy of a large network model. In practice, choosing the appropriate student network based on the constraints of the usage conditions while using knowledge distillation (KD) would be a way to meet practical needs.

Multiple-Stage Knowledge Distillation

Knowledge distillation (KD) is a method in which a teacher network guides the learning of a student network, thereby resulting in an improvement in the performance of the student network. Recent research in this area has concentrated on developing effective definitions of knowledge and efficient methods of knowledge transfer while ignoring the learning ability of the student network. To fully utilize this potential learning ability and improve learning efficiency, this study proposes a multiple-stage KD (MSKD) method that allows students to learn the knowledge delivered by the teacher network in multiple stages. The student network in this method consists of a multi-exit architecture, and the students imitate the output of the teacher network at each exit. The final classification by the student network is achieved through ensemble learning. However, because this results in an unreasonable gap between the number of parameters in the student branch network and those in the teacher branch network, as well as a mismatch in learning capacity between these two networks, we extend the MSKD method to a one-to-one multiple-stage KD method. The experimental results reveal that the proposed method applied to the CIFAR100 and Tiny ImageNet datasets exhibits good performance gain. The proposed method of enhancing KD by changing the style of student learning provides new insight into KD.

Mask-Aware Semi-Supervised Object Detection in Floor Plans

Research has been growing on object detection using semi-supervised methods in past few years. We examine the intersection of these two areas for floor-plan objects to promote the research objective of detecting more accurate objects with less labeled data. The floor-plan objects include different furniture items with multiple types of the same class, and this high inter-class similarity impacts the performance of prior methods. In this paper, we present Mask R-CNN-based semi-supervised approach that provides pixel-to-pixel alignment to generate individual annotation masks for each class to mine the inter-class similarity. The semi-supervised approach has a student–teacher network that pulls information from the teacher network and feeds it to the student network. The teacher network uses unlabeled data to form pseudo-boxes, and the student network uses both label data with the pseudo boxes and labeled data as the ground truth for training. It learns representations of furniture items by combining labeled and label data. On the Mask R-CNN detector with ResNet-101 backbone network, the proposed approach achieves a mAP of 98.8%, 99.7%, and 99.8% with only 1%, 5% and 10% labeled data, respectively. Our experiment affirms the efficiency of the proposed approach, as it outperforms the previous semi-supervised approaches using only 1% of the labels.

DistilProtBert: a distilled protein language model used to distinguish between real proteins and their randomly shuffled counterparts.

Recently, deep learning models, initially developed in the field of natural language processing (NLP), were applied successfully to analyze protein sequences. A major drawback of these models is their size in terms of the number of parameters needed to be fitted and the amount of computational resources they require. Recently, 'distilled' models using the concept of student and teacher networks have been widely used in NLP. Here, we adapted this concept to the problem of protein sequence analysis, by developing DistilProtBert, a distilled version of the successful ProtBert model. Implementing this approach, we reduced the size of the network and the running time by 50%, and the computational resources needed for pretraining by 98% relative to ProtBert model. Using two published tasks, we showed that the performance of the distilled model approaches that of the full model. We next tested the ability of DistilProtBert to distinguish between real and random protein sequences. The task is highly challenging if the composition is maintained on the level of singlet, doublet and triplet amino acids. Indeed, traditional machine-learning algorithms have difficulties with this task. Here, we show that DistilProtBert preforms very well on singlet, doublet and even triplet-shuffled versions of the human proteome, with AUC of 0.92, 0.91 and 0.87, respectively. Finally, we suggest that by examining the small number of false-positive classifications (i.e. shuffled sequences classified as proteins by DistilProtBert), we may be able to identify de novo potential natural-like proteins based on random shuffling of amino acid sequences. https://github.com/yarongef/DistilProtBert.

Diversified branch fusion for self-knowledge distillation

Knowledge distillation improves the performance of a compact student network by adding supervision from a pre-trained cumbersome teacher network during training. To avoid the resource consumption of acquiring an extra teacher network, the self-knowledge distillation designs a multi-branch network architecture with shared layers for teacher and student models, which are trained collaboratively in a one-stage manner. However, this method ignores the knowledge of shallow branches and rarely provides diverse knowledge for effective collaboration of different branches. To solve these two shortcomings, this paper proposes a novel Diversified Branch Fusion approach for Self-Knowledge Distillation (DBFSKD). Firstly, we design lightweight networks for adding to the middle layers of the backbone. They capture discriminative information by global–local attention. Then we introduce a diversity loss between different branches to explore diverse knowledge. Moreover, the diverse knowledge is further integrated to form two knowledge sources by a Selective Feature Fusion (SFF) and a Dynamic Logits Fusion (DLF). Thus, the significant knowledge of shallow branches is efficiently utilized and all branches learn from each other through the fused knowledge sources. Extensive experiments with various backbone structures on four public datasets (CIFAR100, Tiny-ImageNet200, ImageNet, and RAF-DB) show superior performance of the proposed method over other methods. More importantly, the DBFSKD achieves even better performance with fewer resource consumption than the baseline.

B-AT-KD: Binary attention map knowledge distillation

Convolutional neural networks (CNNs) have been extensively used in a number of applications and have shown to be quite effective. As the depth and width of the network expand, not only does prediction accuracy improve, but so does the network’s training complexity. To solve this problem, the knowledge distillation (KD) network, which consists of a teacher and a student network, is established. Furthermore, the widely accessible attention mechanism is important in the KD. Therefore, we build the Binary Attention Map Knowledge Distillation (B-AT-KD) model to enhance the performance of the student network by combining the output of the teacher network and associated attention maps with distinct semantics to supervise the student network. In addition, to improve training results, we propose a new global loss function called KDM-Loss and use hyperparameters search to assign suitable weights. Finally, we compare our B-AT-KD to state-of-the-art KD methods on the CIFAR 10, CIFAR 100, and Mini-ImageNet datasets. Experiments reveal that our proposed approach improves CIFAR 10 precision by 3.98%, Mini-ImageNet accuracy by 2.30%, and CIFAR 100 precision by 2.51% while reducing the number of parameters and computations by more than 50%.

A STUDY ON PROFESSIONAL DEVELOPMENT NEEDS OF EFL TEACHERS AT SOME HIGH SCHOOLS IN THE CENTRAL AREA OF VIETNAM

This study aims to investigate the needs of professional development (PD) of EFL teachers at some high schools inthe central area of Vietnam and the ways they would like to carry out their professional development acitivities. The partiticipants included sixty EFL high school teachers who had teaching experience ranging from 2-30 yearsand their ages from 23 to over 50. Data were collected by means of a questionnaire and an interview protocol.The findings show that most of high school EFL teachers perceived PD as the process that helped teachers todevelop new skills, stay up-to-date on current trends, and advance their career. In particular, participation in a network of teachers was considered to be the most common PD activity needed by all the participants. They also stated that taking part in PD activities served the need of understanding more about the cultures of English - speaking countries in order to convey the lessons to students more accurately and lively. Besides, the results indicate that the need to know how to select appropriate materials and the ways to adapt them contributed to EFL teachers’ PD. From the findings, implications for administrators and teachers are put foward for EFL teachers torealize their needs for improvement.