Teacher Network Research Articles

Audio Representation Learning by Distilling Video as Privileged Information

Deep audio representation learning using multi-modal audio-visual data often leads to a better performance compared to uni-modal approaches. However, in real-world scenarios both modalities are not always available at the time of inference, leading to performance degradation by models trained for multi-modal inference. In this work, we propose a novel approach for deep audio representation learning using audio-visual data when the video modality is absent at inference. For this purpose, we adopt teacher-student knowledge distillation under the framework of learning using privileged information (LUPI). While the previous methods proposed for LUPI use soft-labels generated by the teacher, in our proposed method we use <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">embeddings</i> learned by the teacher to train the student network. We integrate our method in two different settings: sequential data where the features are divided into multiple segments throughout time, and non-sequential data where the entire features are treated as one whole segment. In the non-sequential setting both the teacher and student networks are comprised of an encoder component and a task header. We use the embeddings produced by the encoder component of the teacher to train the encoder of the student, while the task header of the student is trained using ground-truth labels. In the sequential setting, the networks have an additional aggregation component that is placed between the encoder and task header. We use two sets of embeddings produced by the encoder and aggregation component of the teacher to train the student. Similar to the non-sequential setting, the task header of the student network is trained using ground-truth labels. We test our framework on two different audio-visual tasks, namely speaker recognition and speech emotion recognition. Through these experiments we show that by treating the video modality as privileged information for the main goal of audio representation learning, our method results in considerable improvements over sole audio-based recognition as well as prior works that use LUPI.

KL-DNAS: Knowledge Distillation-Based Latency Aware-Differentiable Architecture Search for Video Motion Magnification.

Video motion magnification is the task of making subtle minute motions visible. Many times subtle motion occurs while being invisible to the naked eye, e.g., slight deformations in muscles of an athlete, small vibrations in the objects, microexpression, and chest movement while breathing. Magnification of such small motions has resulted in various applications like posture deformities detection, microexpression recognition, and studying the structural properties. State-of-the-art (SOTA) methods have fixed computational complexity, which makes them less suitable for applications requiring different time constraints, e.g., real-time respiratory rate measurement and microexpression classification. To solve this problem, we propose a knowledge distillation-based latency aware-differentiable architecture search (KL-DNAS) method for video motion magnification. To reduce memory requirements and to improve denoising characteristics, we use a teacher network to search the network by parts using knowledge distillation (KD). Furthermore, search among different receptive fields and multifeature connections are applied for individual layers. Also, a novel latency loss is proposed to jointly optimize the target latency constraint and output quality. We are able to find 2.8 × smaller model than the SOTA method and better motion magnification with lesser distortions. https://github.com/jasdeep-singh-007/KL-DNAS.

Fast and High-Performance Learned Image Compression With Improved Checkerboard Context Model, Deformable Residual Module, and Knowledge Distillation.

Deep learning-based image compression has made great progresses recently. However, some leading schemes use serial context-adaptive entropy model to improve the rate-distortion (R-D) performance, which is very slow. In addition, the complexities of the encoding and decoding networks are quite high and not suitable for many practical applications. In this paper, we propose four techniques to balance the trade-off between the complexity and performance. We first introduce the deformable residual module to remove more redundancies in the input image, thereby enhancing compression performance. Second, we design an improved checkerboard context model with two separate distribution parameter estimation networks and different probability models, which enables parallel decoding without sacrificing the performance compared to the sequential context-adaptive model. Third, we develop a three-pass knowledge distillation scheme to retrain the decoder and entropy coding, and reduce the complexity of the core decoder network, which transfers both the final and intermediate results of the teacher network to the student network to improve its performance. Fourth, we introduce L1 regularization to make the numerical values of the latent representation more sparse, and we only encode non-zero channels in the encoding and decoding process to reduce the bit rate. This also reduces the encoding and decoding time. Experiments show that compared to the state-of-the-art learned image coding scheme, our method can be about 20 times faster in encoding and 70-90 times faster in decoding, and our R-D performance is also 2.3% higher. Our method achieves better rate-distortion performance than classical image codecs including H.266/VVC-intra (4:4:4) and some recent learned methods, as measured by both PSNR and MS-SSIM metrics on the Kodak and Tecnick-40 datasets.

Rich Action-semantic Consistent Knowledge for Early Action Prediction.

Early action prediction (EAP) aims to recognize human actions from a part of action execution in ongoing videos, which is an important task for many practical applications. Most prior works treat partial or full videos as a whole, ignoring rich action knowledge hidden in videos, i.e., semantic consistencies among different partial videos. In contrast, we partition original partial or full videos to form a new series of partial videos and mine the Action-Semantic Consistent Knowledge (ASCK) among these new partial videos evolving in arbitrary progress levels. Moreover, a novel Rich Action-semantic Consistent Knowledge network (RACK) under the teacher-student framework is proposed for EAP. Firstly, we use a two-stream pre-trained model to extract features of videos. Secondly, we treat the RGB or flow features of the partial videos as nodes and their action semantic consistencies as edges. Next, we build a bi-directional semantic graph for the teacher network and a single-directional semantic graph for the student network to model rich ASCK among partial videos. The MSE and MMD losses are incorporated as our distillation loss to enrich the ASCK of partial videos from the teacher to the student network. Finally, we obtain the final prediction by summering the logits of different subnetworks and applying a softmax layer. Extensive experiments and ablative studies have been conducted, demonstrating the effectiveness of modeling rich ASCK for EAP. With the proposed RACK, we have achieved state-of-the-art performance on three benchmarks. The code is available at https://github.com/lily2lab/RACK.git.

Balanced Class-Incremental 3D Object Classification and Retrieval

Most existing 3D object classification and retrieval algorithms rely on one-off supervised learning on closed 3D object sets and tend to provide rigid convolutional neural networks with little scalability. Such limitations substantially restrict their potential to learn newly emerged 3D object classes continually in the real world. Aiming to go beyond these limitations, we innovatively propose two new and challenging tasks: class-incremental 3D object classification ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CI-3DOC</i> ) and class-incremental 3D object retrieval ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CI-3DOR</i> ), the key to which is class-incremental 3D representation learning. It expects the network to update continually to learn new 3D class representations without forgetting the previously learned ones. To this end, we design a novel balanced distillation network <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(BDNet)</i> that uses a dual supervision mechanism to balance between consolidating old knowledge (stability) and adapting to new 3D object classes (plasticity) carefully. On the one hand, we employ stability-based supervision to retain the stable and discriminative information of old classes that greatly benefit both classification and retrieval tasks. On the other hand, we use plasticity-based supervision to improve the network's generalization for learning new class 3D representations by transferring knowledge from a temporary teacher network to the current model. By properly handling the relationship between the two modules, we achieve a surprising performance improvement. Furthermore, considering there is no available dataset for evaluation, we build two 3D datasets, INOR-1 and INOR-2, to evaluate these two new tasks. Extensive experimental results demonstrate that our method can significantly outperform other state-of-the-art class-incremental learning methods. Even if we store 500-1000 fewer 3D objects than SOTA methods, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">BDNet</i> still achieves comparable performance.

A lightweight residual network based on improved knowledge transfer and quantized distillation for cross-domain fault diagnosis of rolling bearings

Predictive maintenance advocates the use of artificial intelligence to analyze big data and provides support for monitoring health conditions and planning maintenance activities in smart manufacturing systems. However, deep networks with massive parameters incur high computational costs, hindering its practical applications. Knowledge distillation (KD) transfers knowledge from a deep network to a lightweight network at the cost of an evident decline in diagnostic accuracy. Instead of an accuracy-size trade-off, a framework for creating a lightweight residual network with higher diagnostic accuracy is proposed in this paper. First, the generalization ability of the teacher network is enhanced based on a domain-adversarial neural network (DANN) with normalization attention mechanism (NAM) and ResNet18. Then, a small student network, ResNet6, efficiently learns the knowledge from its well-learned teacher network through the improved probability KD (IPKD) and uniform quantization. The IPKD is designed to obtain the utmost knowledge, represented as probability distributions of labels and feature space, and narrow the gap between teacher and student networks. Meanwhile, the uniform quantization is incorporated into the distillation process for quantization and distillation co-optimization, further compressing the student network. Experiments on two open-source and one wheelset bearing sets are conducted for performance verification and comparison. The results demonstrate that the obtained lightweight student network has a similar accuracy to the deep network for cross-domain fault diagnosis of bearings with various damages and speeds, even bearing sources, and its quantized network has a much smaller size while retaining a comparatively similar accuracy, which makes it more practical for actual industrial equipment.

Effects of ability grouping on students’ collaborative problem solving patterns: Evidence from lag sequence analysis and epistemic network analysis

Grouping students according to their abilities and maximizing each member's role is a key issue in collaborative learning research. Although some studies have investigated the effects of homogeneous or heterogeneous grouping on collaborative learning performance, the differences between two approaches remains largely unexplored. During the course of an eight-week term, 42 student teachers participated in a multitasking collaborative problem solving activity. This study employed content analysis, Lag Sequence Analysis, and Epistemic Network Analysis to examine the effects of ability grouping on the behavioral patterns and cognitive networks of the student teachers. The coding results show that, compared with the homogenous groups, the heterogeneous groups had more socially oriented communication, and they were also more inclined to propose questions concerning learning task. Students with similar ability were more successful in establishing common representations of problem solving in peer interactions. In addition, the statistical analysis showed that heterogeneous groups had significantly higher learning performance than homogeneous groups. Revealing and comparing the behavioral patterns, heterogeneous groups had more behavioral sequences that were significantly associated with learning performance, which partly explained the significantly higher performance of the heterogeneous group than the homogeneous group. Compared with the homogeneous groups, the most striking difference of the heterogeneous groups was in how they proceeded negotiation. Furthermore, Epistemic Network Analysis detailed how they developed cognitive networks of multitasking collaborative learning from different levels. At the group level, more closely connected CPS skills were observed in the heterogeneous group. At the individual level, three learners in the heterogeneous groups had a distributed discussion focus and trajectory change due to the emerging of student-teacher relationships. In the homogeneous groups, three learners were very close in their discussion focus at beginning and the changes in the trajectories are also very close. Finally, we stated the implications and limitations and pointed out future research directions.

The European List of Key Medicines for Medical Education: A Modified Delphi Study.

Rational prescribing is essential for the quality of health care. However, many final-year medical students and junior doctors lack prescribing competence to perform this task. The availability of a list of medicines that a junior doctor working in Europe should be able to independently prescribe safely and effectively without supervision could support and harmonize teaching and training in clinical pharmacology and therapeutics (CPT) in Europe. Therefore, our aim was to achieve consensus on such a list of medicines that are widely accessible in Europe. For this, we used a modified Delphi study method consisting of three parts. In part one, we created an initial list based on a literature search. In part two, a group of 64 coordinators in CPT education, selected via the Network of Teachers in Pharmacotherapy of the European Association for Clinical Pharmacology and Therapeutics, evaluated the accessibility of each medicine in his or her country, and provided a diverse group of experts willing to participate in the Delphi part. In part three, 463 experts from 24 European countries were invited to participate in a 2-round Delphi study. In total, 187 experts (40%) from 24 countries completed both rounds and evaluated 416 medicines, 98 of which were included in the final list. The top three Anatomical Therapeutic Chemical code groups were (1) cardiovascular system (n = 23), (2) anti-infective (n = 21), and (3) musculoskeletal system (n = 11). This European List of Key Medicines for Medical Education could be a starting point for country-specific lists and could be used for the training and assessment of CPT.

Enhancing deep feature representation in self-knowledge distillation via pyramid feature refinement

In recent years, various self-knowledge distillation approaches have been proposed to reduce the cost of training teacher networks. However, these methods often overlook the significance of deep features. To address this limitation and strengthen the capability of deep features while preserving the ability of shallow features, we propose performing Self-Knowledge Distillation via Pyramid Feature Refinement (PR-SKD). Inspired by the representation learning characteristics of deep neural networks, PR-SKD builds a cohort of sub-networks with a pyramid architecture to hierarchically transfer refined information to the target network. According to the different contributions and functions between deep and shallow feature maps, our PR-SKD fully utilizes feature information to improve deep feature representation ability without compromising the capability of shallow feature maps. Extensive experiments on various image classification datasets demonstrate the superiority of our proposed method over widely used state-of-the-art knowledge distillation methods. The code is available at: https://github.com/wo16pao/PR-SKD.

ABUS tumor segmentation via decouple contrastive knowledge distillation

Objective. In recent years, deep learning-based methods have become the mainstream for medical image segmentation. Accurate segmentation of automated breast ultrasound (ABUS) tumor plays an essential role in computer-aided diagnosis. Existing deep learning models typically require a large number of computations and parameters. Approach. Aiming at this problem, we propose a novel knowledge distillation method for ABUS tumor segmentation. The tumor or non-tumor regions from different cases tend to have similar representations in the feature space. Based on this, we propose to decouple features into positive (tumor) and negative (non-tumor) pairs and design a decoupled contrastive learning method. The contrastive loss is utilized to force the student network to mimic the tumor or non-tumor features of the teacher network. In addition, we designed a ranking loss function based on ranking the distance metric in the feature space to address the problem of hard-negative mining in medical image segmentation. Main results. The effectiveness of our knowledge distillation method is evaluated on the private ABUS dataset and a public hippocampus dataset. The experimental results demonstrate that our proposed method achieves state-of-the-art performance in ABUS tumor segmentation. Notably, after distilling knowledge from the teacher network (3D U-Net), the Dice similarity coefficient (DSC) of the student network (small 3D U-Net) is improved by 7%. Moreover, the DSC of the student network (3D HR-Net) reaches 0.780, which is very close to that of the teacher network, while their parameters are only 6.8% and 12.1% of 3D U-Net, respectively. Significance. This research introduces a novel knowledge distillation method for ABUS tumor segmentation, significantly reducing computational demands while achieving state-of-the-art performance. The method promises enhanced accuracy and feasibility for computer-aided diagnosis in diverse imaging scenarios.

Knowledge Distillation for Traversable Region Detection of LiDAR Scan in Off-Road Environments.

In this study, we propose a knowledge distillation (KD) method for segmenting off-road environment range images. Unlike urban environments, off-road terrains are irregular and pose a higher risk to hardware. Therefore, off-road self-driving systems are required to be computationally efficient. We used LiDAR point cloud range images to address this challenge. The three-dimensional (3D) point cloud data, which are rich in detail, require substantial computational resources. To mitigate this problem, we employ a projection method to convert the image into a two-dimensional (2D) image format using depth information. Our soft label-based knowledge distillation (SLKD) effectively transfers knowledge from a large teacher network to a lightweight student network. We evaluated SLKD using the RELLIS-3D off-road environment dataset, measuring the performance with respect to the mean intersection of union (mIoU) and GPU floating point operations per second (GFLOPS). The experimental results demonstrate that SLKD achieves a favorable trade-off between mIoU and GFLOPS when comparing teacher and student networks. This approach shows promise for enabling efficient off-road autonomous systems with reduced computational costs.

Topological persistence guided knowledge distillation for wearable sensor data

Deep learning methods have achieved a lot of success in various applications involving converting wearable sensor data to actionable health insights. A common application areas is activity recognition, where deep-learning methods still suffer from limitations such as sensitivity to signal quality, sensor characteristic variations, and variability between subjects. To mitigate these issues, robust features obtained by topological data analysis (TDA) have been suggested as a potential solution. However, there are two significant obstacles to using topological features in deep learning: (1) large computational load to extract topological features using TDA, and (2) different signal representations obtained from deep learning and TDA which makes fusion difficult. In this paper, to enable integration of the strengths of topological methods in deep-learning for time-series data, we propose to use two teacher networks — one trained on the raw time-series data, and another trained on persistence images generated by TDA methods. These two teachers are jointly used to distill a single student model, which utilizes only the raw time-series data at test-time. This approach addresses both issues. The use of KD with multiple teachers utilizes complementary information, and results in a compact model with strong supervisory features and an integrated richer representation. To assimilate desirable information from different modalities, we design new constraints, including orthogonality imposed on feature correlation maps for improving feature expressiveness and allowing the student to easily learn from the teacher. Also, we apply an annealing strategy in KD for fast saturation and better accommodation from different features, while the knowledge gap between the teachers and student is reduced. Finally, a robust student model is distilled, which can at test-time uses only the time-series data as an input, while implicitly preserving topological features. The experimental results demonstrate the effectiveness of the proposed method on wearable sensor data. The proposed method shows 71.74% in classification accuracy on GENEActiv with WRN16-1 (1D CNNs) student, which outperforms baselines and takes much less processing time (less than 17 sec) than teachers on 6k testing samples.

Advanced integrated segmentation approach for semi-supervised infrared ship target identification

Infrared ship target segmentation is the important basis of infrared guided weapon in the sea-air context. Typically, accurate infrared ship target segmentation relies on a large number of pixel-level labels. However, it is difficult to obtain them. To this end, we present a method of Semi-supervised Infrared Ship Target Segmentation with Dual Branch (SeISTS-DB), which utilizes a small amount of labeled data and a large amount of unlabeled data to train model and improve segmentation performance. There are three main contributions. First, we design a target segmentation branch to generate the pseudo labels for unlabeled data. It consists of a dual learning network and a segmentation network. The dual learning network generates pseudo labels with weights for unlabeled data. The segmentation network is trained using both labeled data and unlabeled data with pseudo labels to achieve target segmentation of infrared ship, obtaining the preliminary segmentation results. Secondly, we introduce an error segmentation pixel correction branch, which contains a student network and a teacher network, to modify the pixel category error of the preliminary segmentation map. Finally, the outputs of the two branches are combined to obtain the final segmentation result. The SeISTS-DB is compared with other fully-supervised and semi-supervised methods on the infrared ship images dataset. Experimental results demonstrate that when the labeled data accounts for 1/8 of the training data, the mean Intersection over Union (mIou) is respectively improved by 15.35% and 6.19% at most. Besides, it is also compared with other methods on the public IRSTD-1k dataset, when the proportion of labeled images is 1/8, the mIoU is respectively improved by 11.76% at most compared to the state-of-the-art semi-supervised methods, demonstrating its effectiveness.

A teacher–student deep learning strategy for extreme low resolution unsafe action recognition in construction projects

A large proportion of construction accidents are caused by workers’ unsafe actions. Due to the complexity of the work environment and excessive demands of safety supervision on construction sites, many modern information technologies, such as computer vision (CV) technologies, have been increasingly applied and gradually replaced the traditional manual supervision to automatically identify unsafe behaviors from surveillance video data. Current models focus on high resolution (HR) video containing high quality features. However, challenges remain for more specific real-world construction scenarios, including but not limited to lack of high-quality data, far-field video recognition, privacy protection, and limited computing resources. To address the challenges, in this paper, a simple but effective method relying on the Teacher–Student Learning Architecture is proposed, where the teacher network assists the student network in effectively capturing useful features for downstream action recognition from extreme low resolution (eLR) video which can be easily transferred to other tasks. In addition, a Knowledge Learning Model and a similarity loss function are proposed to better guide the training of student network. In numerical experiments and case studies, our proposed training strategy achieves an top-1 accuracy of 28.53% and 71.81% on eLR HMDB dataset and eLR CMA dataset, 5.96% and 3.89% higher than the baseline model. The proposed strategy can make the automatic safety monitoring of construction works more accurate and reliable, thereby effectively reducing the accident rate and management cost. In addition, the performance of proposed strategy depends on efficient knowledge distillation methods, which may inspire future applications of CV-based deep learning models in construction management to achieve higher recognition results while maintaining low computational cost.

Global key knowledge distillation framework

The performance of deep neural networks is typically associated with their depth and width. As the performance of a model improves, it demands greater computational resources and memory. However, self-distillation offers a pathway to augment this performance by internally transferring knowledge. Traditional self-distillation methods lack guidance from teacher networks, thus lacking the ability to obtain correct and key information. In this paper, we propose a new self-distillation framework—Global key knowledge distillation framework (GK-KD). We implement error correction mechanisms to network logit outputs and assess the importance of feature maps through rank to obtain correct and important knowledge. Experiments demonstrate the effectiveness of this framework: on the CIFAR100 dataset, ResNet18 achieved a 4.56% accuracy improvement over the baseline, ResNet152 also achieved a 5.26% accuracy improvement, and for the VGG series, the average accuracy increased by 5.82%, outperforming previous research methods without adding additional computational costs. Further experiments show that this framework can effectively enhance the robustness and calibration of the network.

Unsupervised domain adaptation for brain structure segmentation via mutual information maximization alignment

Brain structure segmentation in Magnetic Resonance image (MRI) plays an important role in the diagnosis of various neuropsychiatric diseases. Recently, the high cost of manual annotation has motivated research in medical image analysis to focus on unsupervised learning methods. As a promising solution, unsupervised domain adaptation (UDA) has received increasing attention in various medical image segmentation tasks. In this work, we propose a novel UDA framework for brain structure segmentation from MRI volumes. Specifically, we design a self-training-based network architecture to learn cross-domain representations, which is achieved by transferring essential knowledge from a well-trained cumbersome teacher network to a non-trained compact student network using distillation learning technology. The pseudo-labels generated by the pre-trained student model are used to supervise the domain adaptation process of the teacher model. Furthermore, we develop a mutual information maximization alignment module to learn domain-invariant features and cross-domain segmentation knowledge, in which contrastive learning techniques are utilized to maximize the correlation between samples. Finally, we build a unified distillation framework by integrating mutual information distillation and output map distillation, correcting pseudo-labels during the self-training stage, and continuously refining the final predictions of compact networks. Extensive experiments conducted on two datasets with vastly different styles and settings demonstrate that our method is effective in improving segmentation performance on unlabeled target images, and outperforms the state-of-the-art domain adaptation approaches by a large margin. The source code is available at https://github.com/huqian999/UDA-MIMA.

Degradation model and attention guided distillation approach for low resolution face recognition

Deep convolution neural networks (CNN) have shown their efficacy in face recognition tasks due to their ability to extract highly discriminant face representations from face images. On high-resolution benchmark datasets, outstanding identification and verification results have been achieved. However, the performance of these algorithms is significantly degraded when tested on low-resolution (LR) images such as those captured from surveillance cameras. A straightforward solution to this problem is to use both high-resolution (HR) images and corresponding down-sampled LR images during training. Although this strategy improves the performance for LR images, it has some limitations. First, there is a significant difference between down-sampled LR images and LR images from surveillance cameras, leading to performance saturation at an earlier stage. Another limitation is the deterioration in the performance of HR images. In this work, solutions to both these limitations are proposed. A degradation model is proposed that synthesizes LR images from corresponding HR, emulating the real-world degradation effects in synthetic data, thus enabling the face recognition system to tolerate various blurry and noisy effects. To address the deterioration in the performance of HR images, an attention-guided distillation is proposed, which utilizes attention maps from convolutional layers in combination with deep features to transfer informative HR features from teacher to student network. The attention maps from the teacher network guide the student network to a better optimum and produce resolution robust face representations. The results of the proposed approach on the popular LR datasets like SCface, Coxface, and PaSC show that it outperforms the recent state-of-the-art (SOTA) techniques by a significant margin demonstrating its effectiveness for different cross-resolution scenarios.

Frame-Level Teacher-Student Learning With Data Privacy for EEG Emotion Recognition.

Recently, electroencephalogram (EEG) emotion recognition has gradually attracted a lot of attention. This brief designs a novel frame-level teacher-student framework with data privacy (FLTSDP) for EEG emotion recognition. The framework first proposes a teacher-student network without prior professional information for automated filtering of useful frame-level features by a gated mechanism and extracting high-level features by using knowledge distillation to capture the results of EEG emotion recognition from a teacher network and student networks. Then, the results from subnetworks are integrated by using the novel decision module, which, motivated by the voting mechanism, adjusts the composition of feature vectors and improves the weight of accurate prediction to optimize the integration effect. During training, an innovative data privacy protection mechanism is applied for avoiding data sharing, where each student network only inherits weights from all trained networks and does not inherit the training dataset. Here, the framework can be repeatedly optimized and improved by only training the next student subnetwork on new EEG signals. Experimental results show that our framework improves the accuracy of EEG emotion recognition by more than 5% and gets state-of-the-art performance for EEG emotion recognition in the subject-independent mode.

STKD: Distilling Knowledge From Synchronous Teaching for Efficient Model Compression.

Knowledge distillation (KD) transfers discriminative knowledge from a large and complex model (known as teacher) to a smaller and faster one (known as student). Existing advanced KD methods, limited to fixed feature extraction paradigms that capture teacher's structure knowledge to guide the training of the student, often fail to obtain comprehensive knowledge to the student. Toward this end, in this article, we propose a new approach, synchronous teaching knowledge distillation (STKD), to integrate online teaching and offline teaching for transferring rich and comprehensive knowledge to the student. In the online learning stage, a blockwise unit is designed to distill the intermediate-level knowledge and high-level knowledge, which can achieve bidirectional guidance of the teacher and student networks. Intermediate-level information interaction provides more supervisory information to the student network and is useful to enhance the quality of final predictions. In the offline learning stage, the STKD approach applies a pretrained teacher to further improve the performance and accelerate the training process by providing prior knowledge. Trained simultaneously, the student learns multilevel and comprehensive knowledge by incorporating online teaching and offline teaching, which combines the advantages of different KD strategies through our STKD method. Experimental results on the SVHN, CIFAR-10, CIFAR-100, and ImageNet ILSVRC 2012 real-world datasets show that the proposed method achieves significant performance improvements compared with the state-of-the-art methods, especially with satisfying accuracy and model size. Code for STKD is provided at https://github.com/nanxiaotong/STKD.

한국 교육 ODA 교사 연수에 참여한 아제르바이잔 교사들의 경험 분석

This paper uses a qualitative paradigm to analyze the experiences and meanings of Azerbaijani teachers who participated in an online-based training organized by KOICA's Education Official Development Assistance (ODA) project. In other words, the study aimed to explore how Azerbaijani teacher training, designed as a learner-participatory program based on Azerbaijani educational cultural characteristics, is experienced and meaningful to Azerbaijani teachers in the source country and to discuss implications for Korean education ODA, which is strengthening its role as a donor country. To this end, in-depth online individual interviews were conducted with 38 Azerbaijani teachers, and the results are presented using the ‘analysis thematique’ method, which categorizes key interview findings into themes. As a result of the study, the narratives of Azerbaijani teachers who participated in KOICA's education ODA can be categorized as 1) improving their self-efficacy in teaching practice, 2) reconstructing their perceptions of teaching and learning, 3) reconstructing their future education and teacher identity, and 4) aspiring to spread their training effectiveness and join international teacher networks. Based on these findings, implications for Korea's education ODA are presented.