Accurate delineation of the prostate and surrounding pelvic structures is critical to successful treatment planning, accurate identification, and staging of prostate cancer. Segmentation of anatomically complex regions surrounding the prostate in CT imaging can be challenging due to low soft-tissue contrast and complex boundary delineations. In this work, we investigate three complementary paradigms of knowledge distillation-voxel-level, region-level, and a dynamically weighted combination of the two-to improve segmentation performance for the prostate and parailiac regions. The region-level approach imposes the semantic coherence of network predictions via the region-wise contrastive form of supervision, whereas the voxel-level distillation provides fine-tuned supervision in terms of Kullback-Leibler divergence on soft probabilistic outputs. We introduce a novel fusion approach that adds uncertainty-aware dynamic weighting, thus allowing the model to adjust the contribution of every distillation loss in an adaptive manner during training, taking advantage of the strengths of both approaches. The distillation methods are implemented within the dual-network architecture in terms of VNet (Milletari et al. in Proc. Int. Conf. 3D Vis. (3DV):565-571, 2016) and 3D-ResVNet (Wang et al. in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)), thus allowing synergistic learning of different architectural biases. Experimental results on both an in-house collected and annotated CT dataset of prostate cancer patients-where parailiac regions and the prostate gland (including seminal vesicles) are manually segmented-and three public benchmark datasets demonstrate that each individual distillation method consistently improves segmentation accuracy over baseline models. These results indicate that the effectiveness of the proposed distillation strategies generalizes across different datasets and anatomical structures, highlighting their robustness and practical applicability. Complementary supervision at voxel and region levels can improve the delineation of complex pelvic structures in CT imaging of prostate cancer.

The proliferation of fake news on digital media has led to concerns about the credibility of information and trust. Conventional rule-based filtering may not be effective against sophisticated fake news. Here we build a fully integrated end-to-end fake news detection system utilising TinyBERT (prajjwal1/bert-tiny), a distilled transformer model. Our system is deployed in Google Colab, and involves data collection from a Kaggle source, data preparation (title-text concatenation), tokenization with the Hugging Face AutoTokenizer and supervised fine-tuning with the Trainer API. We compare our proposed system with a TF-IDF and Logistic Regression based baseline. The model achieves an accuracy of 99% and a macro F1-score of 0.99 with a test set of 8,980 samples. The confusion matrix shows only 9 wrong predictions, showing the power of the proposed method. Our findings show that smaller transformer models can produce high accuracy and be deployed in practice. Index Terms—fake news, TinyBERT, transformers, natural language processing, text classification, knowledge distillation, misinformation

No modality left behind: Adapting to missing modalities via knowledge distillation for brain tumor segmentation.

Leveraging knowledge distillation for lightweight and interpretable deep learning in Ethiopian medicinal plant classification

SpikeBERT: A language spikformer learned from BERT with knowledge distillation.

Existing research on joint kilowatt-negawatt peer-to-peer (P2P) electricity markets predominantly relies on iterative coordination between prosumers and the distribution system operator (DSO). This not only undermines real-time market efficiency but also compromises prosumers’ privacy. Such drawbacks pose significant barriers to the practical deployment of P2P markets, where fast response and strict privacy protection are critical requirements. Therefore, there is a pressing need for a market-clearing design that can simultaneously achieve scalability, efficiency, and privacy preservation. Our paper proposes a novel system-aware and privacy-preserving market-clearing framework based on knowledge distillation. Distribution locational marginal prices (DLMPs) first capture the system operation signals. These DLMPs are embedded as soft constraints into individual prosumer optimization problems via Kullback-Leibler divergence minimization. A single-level and iteration-free market structure can thus be formulated. A dynamic network usage fee mechanism is then developed by integrating electrical distances. The network usage fee is refined through an iterative penalty-based adjustment process to ensure robust cost allocation. To preserve scalability and privacy, a decentralized market-clearing algorithm is finally constructed based on the Alternating Direction Method of Multipliers (ADMM). The exchanged data during iteration is protected with a differential privacy algorithm. Two cases are imported to validate the proposed framework’s ability to enhance social welfare, maintain grid operational feasibility, and guarantee cost recovery.

Enhancing Few-Shot marble slab surface defect detection: A diffusion framework with knowledge distillation and semantic guidance

Prostate cancer is a prevalent and serious health concern, ranking among the most frequently diagnosed cancers and a leading cause of cancer-related deaths in men worldwide. Early detection and accurate diagnosis are crucial for improving patient outcomes by limiting disease progression. Histopathological image analysis remains the gold standard for prostate cancer detection; however, manual interpretation is time-consuming and requires specialized expertise. To address these challenges, this study proposes a hybrid deep learning framework that combines an ensemble of transfer-learned CNNs (VGG-16, DenseNet-121, and AlexNet) with a fine-tuned Vision Transformer (ViT). The CNN ensemble extracts rich local features, while the ViT captures global contextual dependencies through a self-attention mechanism and a multilayer perceptron (MLP). Additionally, a cross-attention fusion (CAF) module integrates local and global features, and knowledge distillation (KD) enables a lightweight student network suitable for efficient clinical deployment. The study utilizes the publicly available PANDA dataset for training and testing. Preprocessing steps, including patch generation, gamma correction, and stain deconvolution, enhance image quality and feature representation. A comprehensive evaluation was conducted using standard performance metrics such as accuracy, true positive rate (TPR), true negative rate (TNR), precision, F1-score, false negative rate (FNR), and false positive rate (FPR). An ablation study confirmed the contribution of each module, highlighting the critical role of ensemble CNNs, CAF, and ViT in improving performance. Experimental results demonstrate that the proposed model outperforms conventional transfer learning models and existing state-of-the-art techniques, achieving 97.91% accuracy, along with significant improvements in TPR, TNR, and reduced FNR/FPR. The computational complexity, evaluated in terms of parameters, FLOPs, GPU memory, and inference time, indicates that the proposed model is more demanding than traditional CNNs. Nevertheless, the architecture strikes a practical balance between predictive accuracy and efficiency, making it suitable for real-world clinical applications. These findings underscore the potential of AI-powered hybrid models in expediting prostate cancer diagnosis and enabling timely intervention for improved patient outcomes.

Abstract Industrial weld seam image segmentation often suffers from accuracy limitations due to neglected linear geometric features and complex background interference. To address this, this paper proposes a weld-aware dual-attention U-Net model, named WADA-UNet. The architecture incorporates a weld-direction-aware spatial attention module and a weld-feature-enhanced channel attention module. These modules interact synergistically through a spatial-channel crossguidance mechanism embedded within the skip connections. Furthermore, an adaptive multi-scale feature fusion module is introduced to dynamically adjust fusion weights according to weld dimensions, thereby improving adaptability to variations in weld width. This paper also adopts a collaborative training strategy to further enhance model performance, which integrates multi-task loss functions, knowledge distillation, and probabilistic calibration. Experimental results show that WADA-UNet achieves a Dice coefficient of 0.9125, a Mean Intersection over Union (MIOU) of 0.9167, and a Mean Pixel Accuracy (MPA) of 0.9592, outperforming all benchmark methods. These results con-firm the model's strong generalization ability and robustness, highlighting its potential as a reliable solution for auto-mated welding quality inspection.

Enhancing end-to-end speech translation via multi-stage knowledge distillation.

Enhancing graph neural networks through universal self-knowledge distillation.

Real-time intrusion detection with millisecond response is critical for Internet of Vehicles (IoV) security but is challenged by extreme class imbalance and high computational costs. This paper proposes a novel multimodal framework integrating Denoising Diffusion Probabilistic Models (DDPM) and Knowledge Distillation (KD). First, multi-source data is transformed into RGB images. A conditional DDPM with timestep and class embeddings balances datasets by generating minority-class samples. The teacher model (DiffuGuardian) fuses text-image features for training. Subsequently, a lightweight student model, LiteSentinel, is designed employing depthwise separable convolutions and inverted residual blocks to reduce parameters. Results on three datasets demonstrate that DiffuGuardian consistently achieves around 98–100% precision, accuracy, recall, and F1-score under 5-fold evaluation, while LiteSentinel maintains approximately 95–99% across all metrics with substantially reduced complexity. DiffuGuardian reaches an inference time of 3.80ms with a model size of 0.10 MB, whereas LiteSentinel further reduces latency to 0.79ms with a size of 0.07 MB, enabling efficient edge deployment for IoV security.

MRKD-PBCL: Multi-level region-wise knowledge distillation and prototype balanced contrastive learning for class incremental semantic segmentation

Knowledge distillation for super-resolution reconstruction and segmentation in forward-facing camera images

PMFM-kdTransformer: An enhanced multi-modal fusion architecture leveraging knowledge distillation for intra-hour solar irradiance prediction

Dual selective gleason pattern-aware multiple instance learning with uncertainty regularization for grade group prediction in histopathology images.

Vision-Language Transformers (VLTs) have achieved remarkable success, yet their high computational costs remain challenging due to numerous input tokens and large model parameters. Existing VLT compression methods primarily rely on single-modality-based token pruning or coarse-grained weight pruning techniques. However, these methods face significant obstacles, such as ignoring the critical alignment of different modalities and lacking layer-wise dynamic token pruning flexibility, exhibiting inevitable performance degradation due to coarsegrained weight pruning, and struggling with the simultaneous compression of both input tokens and model parameters. To address those limitations, we propose MADTP++, a novel approach that integrates custom-made token and weight pruning processes into a unified framework, achieving superior compression in both parameter counts and computational costs. Specifically, for the token pruning process, we introduce the Multi-modality Alignment Guidance (MAG) module and the Dynamic Token Pruning (DTP) module to align semantic features across different modalities and guide the dynamic elimination of redundant tokens based on different input instances. For the weight pruning process, we propose a Hardware-aware Weight Pruning (HWP) module that leverages the Sparse Tensor Cores across diverse hardware setups to enable fine-grained parameter pruning within VLTs. To further unify token and weight pruning, we also propose a Cooperative Optimization Training Strategy that automatically allocates GFLOPs and parameter reductions per branch before pruning and employs Knowledge Distillation Constraints to facilitate joint optimization of both pruning dimensions. Extensive experiments conducted on various VLT models and datasets demonstrate that MADTP++ can significantly reduce model parameters and computational costs while maintaining competitive performance.

Beyond feature mapping: Dual-heterogeneous knowledge distillation with mamba for industrial anomaly detection

Wildfire detection and localization in aerial imagery is critical for rapid response and damage mitigation. Autonomous aerial vehicles (AAVs) enable large area monitoring but face real-time processing challenges due to limited onboard computational and sensor resources. This work introduces a cross-modal knowledge distillation framework for edge-deployed AAVs. A teacher network trained only on thermal images transfers semantic and spatial representations to an optical image based student network when trained in an offline fashion using thermal and optical image pairs. During deployment, the student uses only optical images, thus reducing reliance on multi-sensor payloads while maintaining high detection accuracy. The student model incorporates dual classification heads: an image-level head for fire-free vs. fire-impacted scenes, and a patch-level head for flame vs. no-flame discrimination. This patch-level strategy provides effective fire localization while avoiding the computational overhead of segmentation, making it practical for resource-constrained deployment. Evaluated on aerial wildfire dataset, the framework achieves 90.97% patch-level accuracy, with false alarm and missed detection rates of 8.82% and 14.78%, respectively. The lightweight student model requires only 2.99 GFLOPS with inference time of 0.004s and generates patch-level probability heatmaps for fire region localization. Unlike conventional unimodal systems, this approach leverages thermal-to-optical knowledge transfer to deliver high accuracy, low latency, and precise localization under edge-computing constraints. The code and dataset will be released at https://github.com/medh132/cmkd .

Beyond data dependency: FedPET enables robust federated learning via data-free dual-teacher knowledge distillation