Teacher Network Research Articles

Few-Shot Face Stylization via GAN Prior Distillation.

Face stylization has made notable progress in recent years. However, when training on limited data, the performance of existing approaches significantly declines. Although some studies have attempted to tackle this problem, they either failed to achieve the few-shot setting (less than 10) or can only get suboptimal results. In this article, we propose GAN Prior Distillation (GPD) to enable effective few-shot face stylization. GPD contains two models: a teacher network with GAN Prior and a student network that fulfills end-to-end translation. Specifically, we adapt the teacher network trained on large-scale data in the source domain to the target domain using a handful of samples, where it can learn the target domain's knowledge. Then, we can achieve few-shot augmentation by generating source domain and target domain images simultaneously with the same latent codes. We propose an anchor-based knowledge distillation module that can fully use the difference between the training and the augmented data to distill the knowledge of the teacher network into the student network. The trained student network achieves excellent generalization performance with the absorption of additional knowledge. Qualitative and quantitative experiments demonstrate that our method achieves superior results than state-of-the-art approaches in a few-shot setting.

Toward Generalized Multistage Clustering: Multiview Self-Distillation.

Existing multistage clustering methods independently learn the salient features from multiple views and then perform the clustering task. Particularly, multiview clustering (MVC) has attracted a lot of attention in multiview or multimodal scenarios. MVC aims at exploring common semantics and pseudo-labels from multiple views and clustering in a self-supervised manner. However, limited by noisy data and inadequate feature learning, such a clustering paradigm generates overconfident pseudo-labels that misguide the model to produce inaccurate predictions. Therefore, it is desirable to have a method that can correct this pseudo-label mistraction in multistage clustering to avoid bias accumulation. To alleviate the effect of overconfident pseudo-labels and improve the generalization ability of the model, this article proposes a novel multistage deep MVC framework where multiview self-distillation (DistilMVC) is introduced to distill dark knowledge of label distribution. Specifically, in the feature subspace at different hierarchies, we explore the common semantics of multiple views through contrastive learning and obtain pseudo-labels by maximizing the mutual information between views. Additionally, a teacher network is responsible for distilling pseudo-labels into dark knowledge, supervising the student network and improving its predictive capabilities to enhance its robustness. Extensive experiments on real-world multiview datasets show that our method has better clustering performance than the state-of-the-art (SOTA) methods.

Fast-Vid2Vid++: Spatial-Temporal Distillation for Real-Time Video-to-Video Synthesis.

Video-to-Video synthesis (Vid2Vid) gains remarkable performance in generating a photo-realistic video from a sequence of semantic maps, such as segmentation, sketch and pose. However, this pipeline is heavily limited to high computational cost and long inference latency, mainly attributed to two essential factors: 1) network architecture parameters, 2) sequential data stream. Recently, the parameters of image-based generative models have been significantly reduced via more efficient network architectures. Existing methods mainly focus on slimming network architectures but ignore the size of the sequential data stream. Moreover, due to the lack of temporal coherence, image-based compression is not sufficient for the compression of the video task. In this paper, we present a spatial-temporal hybrid distillation compression framework, Fast-Vid2Vid++, which focuses on knowledge distillation of the teacher network and the data stream of generative models on both space and time. Fast-Vid2Vid++ makes the first attempt at time dimension to transfer hierarchical features and time coherence knowledge to reduce computational resources and accelerate inference. Specifically, we compress the data stream spatially and reduce the temporal redundancy. We distill the knowledge of the hierarchical features and the final response from the teacher network to the student network in high-resolution and full-time domains. We transfer the long-term dependencies of the features and video frames to the student model. After the proposed spatial-temporal hybrid knowledge distillation (Spatial-Temporal-HKD), our model can synthesize high-resolution key-frames using the low-resolution data stream. Finally, Fast-Vid2Vid++ interpolates intermediate frames by motion compensation with slight latency and generates full-length sequences with motion-aware inference (MAI). On standard benchmarks, Fast-Vid2Vid++ achieves a real-time performance of 30-59 FPS and saves 28-35× computational cost on a single V100 GPU. Code and models are publicly available.

Digital Competencies in Education during the 4.0 Revolution

In times of the 4.0 Revolution, digital education is becoming increasingly important as digital skills are crucial for teachers and students. Research shows that most teachers have appropriate competencies in digital technologies, but they lack the motivation to develop them. There are also obstacles to implementing digital education, such as a lack of teaching and technical support. Forms of support for teachers in improving digital competencies are mainly based on training, although there are also self-assessment tools and teacher networks. The conclusion of the research is the need to properly prepare teachers to implement digital education in the context of the 4.0 Rdigitalevolution.

Learning From Teacher’s Failure: A Reflective Learning Paradigm for Knowledge Distillation

Knowledge Distillation transfers knowledge learned by a teacher network to a student network. A common mode of knowledge transfer is directly using the teacher network’s experience for all samples without differentiating whether the experience of teacher is successful or not. According to common sense, experience varies with its nature. Successful experience is used for guidance, and failed experience is used for correction. Inspired by that, this paper analyzes the failure of teacher and proposes a reflective learning paradigm, which additional uses heuristic knowledge extracted from the teacher’s failure besides following the authority of teacher. Specifically, this paper defines Mutual Error Distance (MED) based on the teacher’s wrong predictions. MED measures the adequacy of the decision boundary learned by teacher, which concretizes the failure of teacher. Then, this paper proposes DCGD (divide-and-conquer grouping distillation) to critically transfer the teacher’s knowledge by grouping the target task into small-scale subtasks and designing multi-branch networks on the basis of MED. Finally, a switchable training mechanism is designed to integrate a regular student which provides an option of student network without parameter addition compared with the multi-branch student network. Extensive experiments on three image classification benchmarks (CIFAR-10, CIFAR-100 and TinyImageNet) show the effectiveness of the proposed paradigm. Especially on CIFAR-100 dataset, the average error of students using DCGD+DKD decreased by 4.28%. In addition, the experiment results show that the paradigm is also applicable to self-distillation.

Multi-Task Momentum Distillation for Multimodal Sentiment Analysis

In the field of Multimodal Sentiment Analysis (MSA), the prevailing methods are devoted to developing intricate network architectures to capture the intra- and inter-modal dynamics, which necessitates numerous parameters and poses more difficulties in terms of interpretability in multimodal modeling. Besides, the heterogeneous nature of multiple modalities (text, audio, and vision) introduces significant modality gaps, thereby making multimodal representation learning an ongoing challenge. To address the aforementioned issues, by considering the learning process of modalities as multiple subtasks, we propose a novel approach named Multi-Task Momentum Distillation (MTMD) which succeeds in reducing the gap among different modalities. Specifically, according to the abundance of semantic information, we treat the subtasks of textual and multimodal representations as the teacher networks while the subtasks of acoustic and visual representations as the student ones to present knowledge distillation, which transfers the sentiment-related knowledge guided by the regression and classification subtasks. Additionally, we adopt unimodal momentum models to explore modality-specific knowledge deeply and employ adaptive momentum fusion factors to learn a robust multimodal representation. Furthermore, we provide a theoretical perspective of mutual information maximization by interpreting MTMD as generating sentiment-related views in various ways. Extensive experiments illustrate the superiority of our approach compared with the state-of-the-art methods in MSA.

Unsupervised Vision‐Based Structural Anomaly Detection and Localization with Reverse Knowledge Distillation

Most of vision‐based methods for structural damage detection rely on supervised learning, requiring a substantial number of labeled images for model training, which is labor‐intensive and time‐consuming. To address these challenges, this study introduces a vision‐based structural anomaly detection and localization approach using unsupervised learning and reverse knowledge distillation. The proposed model incorporates a teacher model, a student model, and a trainable one‐class bottleneck embedding module. The asymmetrical architecture of the teacher and student models forms an encoder‐decoder structure for parameter transfer and feature extraction. The student network receives a specific embedding from the teacher network as input and target, facilitating the recovery of multiscale information from the teacher. Training images only contain the undamaged structures, and the teacher model, a pretrained model, instructs the student model to remember their undamaged features to detect and localize damages in unseen testing images. Through experiments, including a comparison among five candidate backbones for pretrained teacher models based on the residual network and testing across various structural damage types, the optimal model is identified, demonstrating good performance in both anomaly detection and localization. Furthermore, the model’s generalization performance is thoroughly validated, confirming its efficacy across diverse scenarios.

LENAS: Learning-Based Neural Architecture Search and Ensemble for 3-D Radiotherapy Dose Prediction.

Radiation therapy treatment planning requires balancing the delivery of the target dose while sparing normal tissues, making it a complex process. To streamline the planning process and enhance its quality, there is a growing demand for knowledge-based planning (KBP). Ensemble learning has shown impressive power in various deep learning tasks, and it has great potential to improve the performance of KBP. However, the effectiveness of ensemble learning heavily depends on the diversity and individual accuracy of the base learners. Moreover, the complexity of model ensembles is a major concern, as it requires maintaining multiple models during inference, leading to increased computational cost and storage overhead. In this study, we propose a novel learning-based ensemble approach named LENAS, which integrates neural architecture search with knowledge distillation for 3-D radiotherapy dose prediction. Our approach starts by exhaustively searching each block from an enormous architecture space to identify multiple architectures that exhibit promising performance and significant diversity. To mitigate the complexity introduced by the model ensemble, we adopt the teacher-student paradigm, leveraging the diverse outputs from multiple learned networks as supervisory signals to guide the training of the student network. Furthermore, to preserve high-level semantic information, we design a hybrid loss to optimize the student network, enabling it to recover the knowledge embedded within the teacher networks. The proposed method has been evaluated on two public datasets: 1) OpenKBP and 2) AIMIS. Extensive experimental results demonstrate the effectiveness of our method and its superior performance to the state-of-the-art methods. Code: github.com/hust-linyi/LENAS.

Restructuring the Teacher and Student in Self-Distillation.

Knowledge distillation aims to achieve model compression by transferring knowledge from complex teacher models to lightweight student models. To reduce reliance on pre-trained teacher models, self-distillation methods utilize knowledge from the model itself as additional supervision. However, their performance is limited by the same or similar network architecture between the teacher and student. In order to increase architecture variety, we propose a new self-distillation framework called restructured self-distillation (RSD), which involves restructuring both the teacher and student networks. The self-distilled model is expanded into a multi-branch topology to create a more powerful teacher. During training, diverse student sub-networks are generated by randomly discarding the teacher's branches. Additionally, the teacher and student models are linked by a randomly inserted feature mixture block, introducing additional knowledge distillation in the mixed feature space. To avoid extra inference costs, the branches of the teacher model are then converted back to its original structure equivalently. Comprehensive experiments have demonstrated the effectiveness of our proposed framework for most architectures on CIFAR-10/100 and ImageNet datasets. Code is available at https://github.com/YujieZheng99/RSD.

The Research Lineage and Knowledge Distillation of the Whole Process “Immersion” Curriculum Civic Teaching

Abstract This research investigates the application of immersive teaching in Civics, enhanced by knowledge distillation techniques. We employed feature distillation and sample-balanced distillation algorithms to refine both the teacher and student networks, ensuring effective content delivery and deeper knowledge acquisition. Experimental results demonstrate that the student networks, optimized with distillation and identification loss, significantly outperform their unoptimized counterparts in feature retention. Furthermore, the integration of IDA data enhancement and kvDKD decoupled knowledge distillation methods substantially boosts the model’s efficiency and accuracy. Our teaching experiments revealed heightened student engagement and learning outcomes in Civics, with all students covering 8 of 12 topics and an average of 27 students scoring above 95 per topic. A comparative analysis with a control group confirms the efficacy of immersive teaching approaches in improving student performance and the quality of Civics education.

Post-Pandemic Analysis on the Factors Affecting Academic Performance of Tourism and Hospitality Management Students During COVID-19 Pandemic: Basis of an Enhanced Learning Continuity Plan

World Health Organization (WHO) Director-General and Committee members declared in 2023 that COVID-19 will no longer constitute a public health emergency (PHEIC). During the pandemic surge in global business, the educational system worldwide, including hospitality education, has been greatly affected by the pandemic crisis. The influence of COVID-19 causes worldwide school closures that have far-reaching implications for students’ short- and long-term growth because of the disruption of teacher and student networks, leading to poor performance. Thus, the study aims to analyze the factors that affect the academic performance of Tourism and Hospitality Management students of Far Eastern University Manila, Philippines, enrolled in the academic year 2020-2021 during the COVID-19 surge. The study used a descriptive survey design using a self-administered survey instrument with 342 respondents. The findings of the study showed that different factors in technology and learning management system (LMS), student’s environment, study area, delivery of academic subjects offered by the teachers, and the assessment used by the school during the distance learning have impacted the academic performance of students as evidence in the decline of Quality Point Average (QPA) on both programs from the Institute of Tourism and Hospitality Management despite that distance learning is designed to be convenient and easy. The factors influencing academic performance, such as technology and study environment, primarily play a significant and considerable role in how students learn and cope with the distant learning mode. The result of the study will serve as a guide in enhancing the University’s continuity plan for flexible learning options, given the factors that need reinforcement to the scholarly performance of the students. The study can be a benchmarking tool among Higher Education Institutions (HEIs) in the Philippines as a basis to further strengthen their learning strategies relative to flexible teaching and learning modalities, refining the scholarly learning assessment tools as well as sustaining the schools’ continuity framework and achieving the academic goals and aspiration for the success of students and graduates of the HEIs whether in the new normal or the future global crisis.

Audio Representation Learning by Distilling Video as Privileged Information

Deep audio representation learning using multi-modal audio-visual data often leads to a better performance compared to uni-modal approaches. However, in real-world scenarios both modalities are not always available at the time of inference, leading to performance degradation by models trained for multi-modal inference. In this work, we propose a novel approach for deep audio representation learning using audio-visual data when the video modality is absent at inference. For this purpose, we adopt teacher-student knowledge distillation under the framework of learning using privileged information (LUPI). While the previous methods proposed for LUPI use soft-labels generated by the teacher, in our proposed method we use <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">embeddings</i> learned by the teacher to train the student network. We integrate our method in two different settings: sequential data where the features are divided into multiple segments throughout time, and non-sequential data where the entire features are treated as one whole segment. In the non-sequential setting both the teacher and student networks are comprised of an encoder component and a task header. We use the embeddings produced by the encoder component of the teacher to train the encoder of the student, while the task header of the student is trained using ground-truth labels. In the sequential setting, the networks have an additional aggregation component that is placed between the encoder and task header. We use two sets of embeddings produced by the encoder and aggregation component of the teacher to train the student. Similar to the non-sequential setting, the task header of the student network is trained using ground-truth labels. We test our framework on two different audio-visual tasks, namely speaker recognition and speech emotion recognition. Through these experiments we show that by treating the video modality as privileged information for the main goal of audio representation learning, our method results in considerable improvements over sole audio-based recognition as well as prior works that use LUPI.

KL-DNAS: Knowledge Distillation-Based Latency Aware-Differentiable Architecture Search for Video Motion Magnification.

Video motion magnification is the task of making subtle minute motions visible. Many times subtle motion occurs while being invisible to the naked eye, e.g., slight deformations in muscles of an athlete, small vibrations in the objects, microexpression, and chest movement while breathing. Magnification of such small motions has resulted in various applications like posture deformities detection, microexpression recognition, and studying the structural properties. State-of-the-art (SOTA) methods have fixed computational complexity, which makes them less suitable for applications requiring different time constraints, e.g., real-time respiratory rate measurement and microexpression classification. To solve this problem, we propose a knowledge distillation-based latency aware-differentiable architecture search (KL-DNAS) method for video motion magnification. To reduce memory requirements and to improve denoising characteristics, we use a teacher network to search the network by parts using knowledge distillation (KD). Furthermore, search among different receptive fields and multifeature connections are applied for individual layers. Also, a novel latency loss is proposed to jointly optimize the target latency constraint and output quality. We are able to find 2.8 × smaller model than the SOTA method and better motion magnification with lesser distortions. https://github.com/jasdeep-singh-007/KL-DNAS.

Fast and High-Performance Learned Image Compression With Improved Checkerboard Context Model, Deformable Residual Module, and Knowledge Distillation.

Deep learning-based image compression has made great progresses recently. However, some leading schemes use serial context-adaptive entropy model to improve the rate-distortion (R-D) performance, which is very slow. In addition, the complexities of the encoding and decoding networks are quite high and not suitable for many practical applications. In this paper, we propose four techniques to balance the trade-off between the complexity and performance. We first introduce the deformable residual module to remove more redundancies in the input image, thereby enhancing compression performance. Second, we design an improved checkerboard context model with two separate distribution parameter estimation networks and different probability models, which enables parallel decoding without sacrificing the performance compared to the sequential context-adaptive model. Third, we develop a three-pass knowledge distillation scheme to retrain the decoder and entropy coding, and reduce the complexity of the core decoder network, which transfers both the final and intermediate results of the teacher network to the student network to improve its performance. Fourth, we introduce L1 regularization to make the numerical values of the latent representation more sparse, and we only encode non-zero channels in the encoding and decoding process to reduce the bit rate. This also reduces the encoding and decoding time. Experiments show that compared to the state-of-the-art learned image coding scheme, our method can be about 20 times faster in encoding and 70-90 times faster in decoding, and our R-D performance is also 2.3% higher. Our method achieves better rate-distortion performance than classical image codecs including H.266/VVC-intra (4:4:4) and some recent learned methods, as measured by both PSNR and MS-SSIM metrics on the Kodak and Tecnick-40 datasets.

Rich Action-semantic Consistent Knowledge for Early Action Prediction.

Early action prediction (EAP) aims to recognize human actions from a part of action execution in ongoing videos, which is an important task for many practical applications. Most prior works treat partial or full videos as a whole, ignoring rich action knowledge hidden in videos, i.e., semantic consistencies among different partial videos. In contrast, we partition original partial or full videos to form a new series of partial videos and mine the Action-Semantic Consistent Knowledge (ASCK) among these new partial videos evolving in arbitrary progress levels. Moreover, a novel Rich Action-semantic Consistent Knowledge network (RACK) under the teacher-student framework is proposed for EAP. Firstly, we use a two-stream pre-trained model to extract features of videos. Secondly, we treat the RGB or flow features of the partial videos as nodes and their action semantic consistencies as edges. Next, we build a bi-directional semantic graph for the teacher network and a single-directional semantic graph for the student network to model rich ASCK among partial videos. The MSE and MMD losses are incorporated as our distillation loss to enrich the ASCK of partial videos from the teacher to the student network. Finally, we obtain the final prediction by summering the logits of different subnetworks and applying a softmax layer. Extensive experiments and ablative studies have been conducted, demonstrating the effectiveness of modeling rich ASCK for EAP. With the proposed RACK, we have achieved state-of-the-art performance on three benchmarks. The code is available at https://github.com/lily2lab/RACK.git.

Balanced Class-Incremental 3D Object Classification and Retrieval

Most existing 3D object classification and retrieval algorithms rely on one-off supervised learning on closed 3D object sets and tend to provide rigid convolutional neural networks with little scalability. Such limitations substantially restrict their potential to learn newly emerged 3D object classes continually in the real world. Aiming to go beyond these limitations, we innovatively propose two new and challenging tasks: class-incremental 3D object classification ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CI-3DOC</i> ) and class-incremental 3D object retrieval ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CI-3DOR</i> ), the key to which is class-incremental 3D representation learning. It expects the network to update continually to learn new 3D class representations without forgetting the previously learned ones. To this end, we design a novel balanced distillation network <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(BDNet)</i> that uses a dual supervision mechanism to balance between consolidating old knowledge (stability) and adapting to new 3D object classes (plasticity) carefully. On the one hand, we employ stability-based supervision to retain the stable and discriminative information of old classes that greatly benefit both classification and retrieval tasks. On the other hand, we use plasticity-based supervision to improve the network's generalization for learning new class 3D representations by transferring knowledge from a temporary teacher network to the current model. By properly handling the relationship between the two modules, we achieve a surprising performance improvement. Furthermore, considering there is no available dataset for evaluation, we build two 3D datasets, INOR-1 and INOR-2, to evaluate these two new tasks. Extensive experimental results demonstrate that our method can significantly outperform other state-of-the-art class-incremental learning methods. Even if we store 500-1000 fewer 3D objects than SOTA methods, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">BDNet</i> still achieves comparable performance.

A lightweight residual network based on improved knowledge transfer and quantized distillation for cross-domain fault diagnosis of rolling bearings

Predictive maintenance advocates the use of artificial intelligence to analyze big data and provides support for monitoring health conditions and planning maintenance activities in smart manufacturing systems. However, deep networks with massive parameters incur high computational costs, hindering its practical applications. Knowledge distillation (KD) transfers knowledge from a deep network to a lightweight network at the cost of an evident decline in diagnostic accuracy. Instead of an accuracy-size trade-off, a framework for creating a lightweight residual network with higher diagnostic accuracy is proposed in this paper. First, the generalization ability of the teacher network is enhanced based on a domain-adversarial neural network (DANN) with normalization attention mechanism (NAM) and ResNet18. Then, a small student network, ResNet6, efficiently learns the knowledge from its well-learned teacher network through the improved probability KD (IPKD) and uniform quantization. The IPKD is designed to obtain the utmost knowledge, represented as probability distributions of labels and feature space, and narrow the gap between teacher and student networks. Meanwhile, the uniform quantization is incorporated into the distillation process for quantization and distillation co-optimization, further compressing the student network. Experiments on two open-source and one wheelset bearing sets are conducted for performance verification and comparison. The results demonstrate that the obtained lightweight student network has a similar accuracy to the deep network for cross-domain fault diagnosis of bearings with various damages and speeds, even bearing sources, and its quantized network has a much smaller size while retaining a comparatively similar accuracy, which makes it more practical for actual industrial equipment.

Effects of ability grouping on students’ collaborative problem solving patterns: Evidence from lag sequence analysis and epistemic network analysis

Grouping students according to their abilities and maximizing each member's role is a key issue in collaborative learning research. Although some studies have investigated the effects of homogeneous or heterogeneous grouping on collaborative learning performance, the differences between two approaches remains largely unexplored. During the course of an eight-week term, 42 student teachers participated in a multitasking collaborative problem solving activity. This study employed content analysis, Lag Sequence Analysis, and Epistemic Network Analysis to examine the effects of ability grouping on the behavioral patterns and cognitive networks of the student teachers. The coding results show that, compared with the homogenous groups, the heterogeneous groups had more socially oriented communication, and they were also more inclined to propose questions concerning learning task. Students with similar ability were more successful in establishing common representations of problem solving in peer interactions. In addition, the statistical analysis showed that heterogeneous groups had significantly higher learning performance than homogeneous groups. Revealing and comparing the behavioral patterns, heterogeneous groups had more behavioral sequences that were significantly associated with learning performance, which partly explained the significantly higher performance of the heterogeneous group than the homogeneous group. Compared with the homogeneous groups, the most striking difference of the heterogeneous groups was in how they proceeded negotiation. Furthermore, Epistemic Network Analysis detailed how they developed cognitive networks of multitasking collaborative learning from different levels. At the group level, more closely connected CPS skills were observed in the heterogeneous group. At the individual level, three learners in the heterogeneous groups had a distributed discussion focus and trajectory change due to the emerging of student-teacher relationships. In the homogeneous groups, three learners were very close in their discussion focus at beginning and the changes in the trajectories are also very close. Finally, we stated the implications and limitations and pointed out future research directions.

The European List of Key Medicines for Medical Education: A Modified Delphi Study.

Rational prescribing is essential for the quality of health care. However, many final-year medical students and junior doctors lack prescribing competence to perform this task. The availability of a list of medicines that a junior doctor working in Europe should be able to independently prescribe safely and effectively without supervision could support and harmonize teaching and training in clinical pharmacology and therapeutics (CPT) in Europe. Therefore, our aim was to achieve consensus on such a list of medicines that are widely accessible in Europe. For this, we used a modified Delphi study method consisting of three parts. In part one, we created an initial list based on a literature search. In part two, a group of 64 coordinators in CPT education, selected via the Network of Teachers in Pharmacotherapy of the European Association for Clinical Pharmacology and Therapeutics, evaluated the accessibility of each medicine in his or her country, and provided a diverse group of experts willing to participate in the Delphi part. In part three, 463 experts from 24 European countries were invited to participate in a 2-round Delphi study. In total, 187 experts (40%) from 24 countries completed both rounds and evaluated 416 medicines, 98 of which were included in the final list. The top three Anatomical Therapeutic Chemical code groups were (1) cardiovascular system (n = 23), (2) anti-infective (n = 21), and (3) musculoskeletal system (n = 11). This European List of Key Medicines for Medical Education could be a starting point for country-specific lists and could be used for the training and assessment of CPT.

Enhancing deep feature representation in self-knowledge distillation via pyramid feature refinement

In recent years, various self-knowledge distillation approaches have been proposed to reduce the cost of training teacher networks. However, these methods often overlook the significance of deep features. To address this limitation and strengthen the capability of deep features while preserving the ability of shallow features, we propose performing Self-Knowledge Distillation via Pyramid Feature Refinement (PR-SKD). Inspired by the representation learning characteristics of deep neural networks, PR-SKD builds a cohort of sub-networks with a pyramid architecture to hierarchically transfer refined information to the target network. According to the different contributions and functions between deep and shallow feature maps, our PR-SKD fully utilizes feature information to improve deep feature representation ability without compromising the capability of shallow feature maps. Extensive experiments on various image classification datasets demonstrate the superiority of our proposed method over widely used state-of-the-art knowledge distillation methods. The code is available at: https://github.com/wo16pao/PR-SKD.