Distillation Scheme Research Articles

A gradual self distillation network with adaptive channel attention for facial expression recognition

Facial expression recognition (FER) is widely applied in various real-world applications, such as security, human–computer interaction, and healthcare, leading to high demands for fast and accurate FER techniques. However, it remains a challenging task to design effective and efficient FER techniques that meet the requirements of real-world applications. In this paper, we propose a lightweight Gradual Self Distillation Network with adaptive channel attention (GSDNet) for accurate and efficient FER. Especially, we propose a novel gradual self distillation strategy which enables the network to learn from itself in a gradual and adaptive way. Specifically, the proposed GSDNet consists of a feature extraction backbone with multiple basic blocks. We plug an adaptive classifier after each basic block. Every two neighbor classifiers form “student & teacher” relationship for gradual knowledge distillation. In particular, the gradual self distillation strategy enables the transfer of key knowledge from deep to shallow layers gradually. Besides, an Adaptive Channel Attention Module (ACAM) is designed to enhance the representation capability of each block for adaptively capturing important features and achieve better FER performance. Extensive experiments on three real-world datasets show that the proposed method GSDNet outperforms the baselines, including state-of-the-art methods. Specifically, the accuracy of GSDNet on the RAF-DB, Affect-net, and FERPlus datasets is 90.91%, 66.11%, and 90.32%, separately. The code is available at https://github.com/Emy-cv/GSDNet.

Joint dual-teacher distillation and unsupervised fusion for unpaired real-world image dehazing

Existing learning-based dehazing algorithms struggle to deal with real world hazy images for lack of paired clean data. Moreover, most dehazing methods require significant computation and memory. To address the above problems, we propose a joint dual-teacher knowledge distillation and unsupervised fusion framework for single image dehazing in this paper. First, considering the complex degradation factors in real-world hazy images, two synthetic-to-real dehazing networks are explored to generate two preliminary dehazing results with the heterogeneous distillation strategy. Second, to get more qualified ground truth, an unsupervised adversarial fusion network is proposed to refine the preliminary outputs of teachers with unpaired clean images. In particular, the unpaired clean images are enhanced to deal with the dim artifacts. Furthermore, to alleviate the structure distortion in the unsupervised adversarial training, we constructed an intermediate image to constrain the output of the fusion network. Finally, considering the memory storage and computation overhead, an end-to-end lightweight student network is trained to learn the mapping from the original hazy image to the output of the fusion network. Experimental results demonstrate that the proposed method achieves state-of-the-art performance on real-world hazy images in terms of no-reference image quality assessment and the parameters.

Self-Supervised 3-D Semantic Representation Learning for Vision-and-Language Navigation.

In vision-and-language navigation (VLN) tasks, most current methods primarily utilize RGB images, overlooking the rich 3-D semantic data inherent to environments. To rectify this, we introduce a novel VLN framework that integrates 3-D semantic information into the navigation process. Our approach features a self-supervised training scheme that incorporates voxel-level 3-D semantic reconstruction to create a detailed 3-D semantic representation. A key component of this framework is a pretext task focused on region queries, which determines the presence of objects in specific 3-D areas. Following this, we devise an long short-term memory (LSTM)-based navigation model that is trained using our 3-D semantic representations. To maximize the utility of these 3-D semantic representations, we implement a cross-modal distillation strategy. This strategy encourages the RGB model's outputs to emulate those from the 3-D semantic feature network, enabling the concurrent training of both branches to merge RGB and 3-D semantic data effectively. Comprehensive evaluations on both the R2R and R4R datasets reveal that our method significantly enhances performance in VLN tasks.

Multimodal MRI segmentation of key structures for microvascular decompression via knowledge-driven mutual distillation and topological constraints.

Microvascular decompression (MVD) is a widely used neurosurgical intervention for the treatment of cranial nerves compression. Segmentation of MVD-related structures, including the brainstem, nerves, arteries, and veins, is critical for preoperative planning and intraoperative decision-making. Automatically segmenting structures related to MVD is still challenging for current methods due to the limited information from a single modality and the complex topology of vessels and nerves. Considering that it is hard to distinguish MVD-related structures, especially for nerve and vessels with similar topology, we design a multimodal segmentation network with a shared encoder-dual decoder structure and propose a clinical knowledge-driven distillation scheme, allowing reliable knowledge transferred from each decoder to the other. Besides, we introduce a class-wise contrastive module to learn the discriminative representations by maximizing the distance among classes across modalities. Then, a projected topological loss based on persistent homology is proposed to constrain topological continuity. We evaluate the performance of our method on in-house dataset consisting of 100 paired HR-T2WI and 3D TOF-MRA volumes. Experiments indicate that our model outperforms the SOTA in DSC by 1.9% for artery, 3.3% for vein and 0.5% for nerve. Visualization results show our method attains improved continuity and less breakage, which is also consistent with intraoperative images. Our method can comprehensively extract the distinct features from multimodal data to segment the MVD-related key structures and preserve the topological continuity, allowing surgeons precisely perceiving the patient-specific target anatomy and substantially reducing the workload of surgeons in the preoperative planning stage. Our resources will be publicly available at https://github.com/JaronTu/Multimodal_MVD_Seg .

Collaborative Knowledge Distillation via Multiknowledge Transfer.

Knowledge distillation (KD), as an efficient and effective model compression technique, has received considerable attention in deep learning. The key to its success is about transferring knowledge from a large teacher network to a small student network. However, most existing KD methods consider only one type of knowledge learned from either instance features or relations via a specific distillation strategy, failing to explore the idea of transferring different types of knowledge with different distillation strategies. Moreover, the widely used offline distillation also suffers from a limited learning capacity due to the fixed large-to-small teacher-student architecture. In this article, we devise a collaborative KD via multiknowledge transfer (CKD-MKT) that prompts both self-learning and collaborative learning in a unified framework. Specifically, CKD-MKT utilizes a multiple knowledge transfer framework that assembles self and online distillation strategies to effectively: 1) fuse different kinds of knowledge, which allows multiple students to learn knowledge from both individual instances and instance relations, and 2) guide each other by learning from themselves using collaborative and self-learning. Experiments and ablation studies on six image datasets demonstrate that the proposed CKD-MKT significantly outperforms recent state-of-the-art methods for KD.

MHD-Net: Memory-Aware Hetero-Modal Distillation Network for Thymic Epithelial Tumor Typing With Missing Pathology Modality.

Fusing multi-modal radiology and pathology data with complementary information can improve the accuracy of tumor typing. However, collecting pathology data is difficult since it is high-cost and sometimes only obtainable after the surgery, which limits the application of multi-modal methods in diagnosis. To address this problem, we propose comprehensively learning multi-modal radiology-pathology data in training, and only using uni-modal radiology data in testing. Concretely, a Memory-aware Hetero-modal Distillation Network (MHD-Net) is proposed, which can distill well-learned multi-modal knowledge with the assistance of memory from the teacher to the student. In the teacher, to tackle the challenge in hetero-modal feature fusion, we propose a novel spatial-differentiated hetero-modal fusion module (SHFM) that models spatial-specific tumor information correlations across modalities. As only radiology data is accessible to the student, we store pathology features in the proposed contrast-boosted typing memory module (CTMM) that achieves type-wise memory updating and stage-wise contrastive memory boosting to ensure the effectiveness and generalization of memory items. In the student, to improve the cross-modal distillation, we propose a multi-stage memory-aware distillation (MMD) scheme that reads memory-aware pathology features from CTMM to remedy missing modal-specific information. Furthermore, we construct a Radiology-Pathology Thymic Epithelial Tumor (RPTET) dataset containing paired CT and WSI images with annotations. Experiments on the RPTET and CPTAC-LUAD datasets demonstrate that MHD-Net significantly improves tumor typing and outperforms existing multi-modal methods on missing modality situations.

Coordinate Attention Guided Dual-Teacher Adaptive Knowledge Distillation for image classification

Knowledge distillation (KD) refers to transferring the knowledge learned from a teacher network with complex architecture and strong learning ability to another student network with light-weight and weak learning ability through a specific distillation strategy. However, most existing KD approaches to image classification often employ a single teacher network to guide the training of the student network. When the teacher network makes an erroneous prediction, the transferred knowledge will deteriorate the performance of the student network. To address or mitigate the above issue, we develop a novel KD approach called Coordinate Attention Guided Dual-Teacher Adaptive Knowledge Distillation (CAG-DAKD), to deliver more discriminative and comprehensive knowledge obtained from two teacher networks to a compact student network. Specifically, we integrate the positive prediction distribution of two teacher networks according to whether the two teacher networks predict correctly and the magnitude of the cross-entropy to deliver better output distribution to guide the student network. Furthermore, to distill the most valuable knowledge from the first teacher network that has a similar architecture to the student network, a coordinate attention mechanism is introduced into different layers of the first teacher network so that the student network can effectively learn more discriminative feature representations. We conduct extensive experiments on three standard image classification datasets: CIFAR10, CIFAR100, and ImageNet to verify the superiority of the proposed method over other state-of-the-art competitors. Code will be available at https://github.com/mdt1219/CAG-DAKD.git/

Holistic-Guided Disentangled Learning With Cross-Video Semantics Mining for Concurrent First-Person and Third-Person Activity Recognition.

The popularity of wearable devices has increased the demands for the research on first-person activity recognition. However, most of the current first-person activity datasets are built based on the assumption that only the human-object interaction (HOI) activities, performed by the camera-wearer, are captured in the field of view. Since humans live in complicated scenarios, in addition to the first-person activities, it is likely that third-person activities performed by other people also appear. Analyzing and recognizing these two types of activities simultaneously occurring in a scene is important for the camera-wearer to understand the surrounding environments. To facilitate the research on concurrent first- and third-person activity recognition (CFT-AR), we first created a new activity dataset, namely PolyU concurrent first- and third-person (CFT) Daily, which exhibits distinct properties and challenges, compared with previous activity datasets. Since temporal asynchronism and appearance gap usually exist between the first- and third-person activities, it is crucial to learn robust representations from all the activity-related spatio-temporal positions. Thus, we explore both holistic scene-level and local instance-level (person-level) features to provide comprehensive and discriminative patterns for recognizing both first- and third-person activities. On the one hand, the holistic scene-level features are extracted by a 3-D convolutional neural network, which is trained to mine shared and sample-unique semantics between video pairs, via two well-designed attention-based modules and a self-knowledge distillation (SKD) strategy. On the other hand, we further leverage the extracted holistic features to guide the learning of instance-level features in a disentangled fashion, which aims to discover both spatially conspicuous patterns and temporally varied, yet critical, cues. Experimental results on the PolyU CFT Daily dataset validate that our method achieves the state-of-the-art performance.

ESGN: Efficient Stereo Geometry Network for Fast 3D Object Detection

Fast stereo based 3D object detectors have made great progress recently. However, they suffer from the inferior accuracy. We argue that the main reason is due to the poor geometry-aware feature representation in 3D space. To solve this problem, we propose an efficient stereo geometry network (ESGN). The key in our ESGN is an efficient geometry-aware feature generation (EGFG) module. Our EGFG module first uses a stereo correlation and reprojection module to construct multi-scale stereo volumes in camera frustum space, second employs a multi-scale bird&#x2019;s eye view (BEV) projection and fusion module to generate multiple geometry-aware features. In these two steps, we adopt deep multi-scale information fusion for discriminative geometry-aware feature generation, without any complex aggregation networks. In addition, we introduce a deep geometry-aware feature distillation scheme to guide stereo feature learning with a LiDAR-based detector. The experiments are performed on the classical KITTI dataset. On KITTI test set, our ESGN outperforms the fast state-of-art-art detector YOLOStereo3D by 5.14% on mAP_3<i>d</i> at 62<i>ms</i>. To the best of our knowledge, our ESGN achieves a best trade-off between accuracy and speed. We hope that our efficient stereo geometry network can provide more possible directions for fast 3D object detection.

Applicability exploration and sustainable assessment of heat integration and vapor recompression heat pump to side-stream extractive distillation processes for separating ternary azeotropic system

The side-stream extractive distillation (SSED) process is attracting extensive attention due to its high energy efficiency for separating ternary azeotropic mixture. However, the correlation between the bottom temperature level of distillation columns depending on the flowrates and compositions of side streams and the introduction of heat integration (HI) and vapor recompression heat pump (VRHP) technology is ignored. In the work, a range of initialization extractive distillation processes including one conventional extractive distillation (CED) process and five SSED processes considering different flowrates, compositions, and feed locations of side streams are proposed for separating acetonitrile/isopropanol/water ternary azeotropic system. The optimization results with the minimum the total annual costs (TAC) as the optimization objective show that not all SSED schemes are superior to the CED separation process, and the SSED separation schemes can achieve up to a 2.72 % reduction in the TAC. Following that, the HI and VRHP assisted extractive distillation schemes are proposed based on the temperature levels between the top and bottom of different distillation columns and compressor restraints for the initialization extractive distillation schemes mentioned after optimization. Moreover, sustainable evaluations covering economy, energy consumption, environment, and energy utilization are implemented for the extractive distillation schemes proposed. The evaluation results indicate that the CED-HI scheme overcomes all SSED-HI and SSED-VRHP schemes in terms of economic costs, but the optimal SSED-HI processes can reduce 8.77 % of energy consumption and 4.73 % of CO2 emissions, while the energy utilization raised from 1.70 % to 1.78 % compared to the CED-HI process.

UniADS: Universal Architecture-Distiller Search for Distillation Gap

In this paper, we present UniADS, the first Universal Architecture-Distiller Search framework for co-optimizing student architecture and distillation policies. Teacher-student distillation gap limits the distillation gains. Previous approaches seek to discover the ideal student architecture while ignoring distillation settings. In UniADS, we construct a comprehensive search space encompassing an architectural search for student models, knowledge transformations in distillation strategies, distance functions, loss weights, and other vital settings. To efficiently explore the search space, we utilize the NSGA-II genetic algorithm for better crossover and mutation configurations and employ the Successive Halving algorithm for search space pruning, resulting in improved search efficiency and promising results. Extensive experiments are performed on different teacher-student pairs using CIFAR-100 and ImageNet datasets. The experimental results consistently demonstrate the superiority of our method over existing approaches. Furthermore, we provide a detailed analysis of the search results, examining the impact of each variable and extracting valuable insights and practical guidance for distillation design and implementation.

SimDistill: Simulated Multi-Modal Distillation for BEV 3D Object Detection

Multi-view camera-based 3D object detection has become popular due to its low cost, but accurately inferring 3D geometry solely from camera data remains challenging and may lead to inferior performance. Although distilling precise 3D geometry knowledge from LiDAR data could help tackle this challenge, the benefits of LiDAR information could be greatly hindered by the significant modality gap between different sensory modalities. To address this issue, we propose a Simulated multi-modal Distillation (SimDistill) method by carefully crafting the model architecture and distillation strategy. Specifically, we devise multi-modal architectures for both teacher and student models, including a LiDAR-camera fusion-based teacher and a simulated fusion-based student. Owing to the ``identical'' architecture design, the student can mimic the teacher to generate multi-modal features with merely multi-view images as input, where a geometry compensation module is introduced to bridge the modality gap. Furthermore, we propose a comprehensive multi-modal distillation scheme that supports intra-modal, cross-modal, and multi-modal fusion distillation simultaneously in the Bird's-eye-view space. Incorporating them together, our SimDistill can learn better feature representations for 3D object detection while maintaining a cost-effective camera-only deployment. Extensive experiments validate the effectiveness and superiority of SimDistill over state-of-the-art methods, achieving an improvement of 4.8% mAP and 4.1% NDS over the baseline detector. The source code will be released at https://github.com/ViTAE-Transformer/SimDistill.

RSKD: Enhanced medical image segmentation via multi-layer, rank-sensitive knowledge distillation in Vision Transformer models

Medical image segmentation is crucial for enhancing diagnostic accuracy through pixel labeling. State-of-the-art networks, despite their performance, have high computational demands, limiting real-time use on constrained devices. Lightweight networks face challenges in balancing detail processing with precision. Vision Transformer models, while promising, also have computational concerns. This study presents a novel method that merges Vision Transformer strengths with a unique knowledge distillation technique. A pivotal element of our approach is the Token Importance Ranking Distillation, which facilitates the meticulous transfer of top-k token importance rankings between a complex teacher model and a simplified student model, guided by a specialized ranking loss function. This method is essential for optimizing the student model to effectively emulate the teacher model’s ability to encapsulate vital semantic and spatial information. Additionally, we introduce an innovative methodology in structural texture knowledge, utilizing a Contourlet Decomposition Module (CDM), which enriches the models with nuanced texture representation, crucial for extracting directional features and capturing intricate global and local contexts in medical imaging. Complementing this, we deploy a unique multi-stage distillation strategy, the Space Channel Cascade Fusion (SCCF), to refine both spatial and channel information concurrently, mitigating redundancy and enhancing representational effectiveness in feature maps. Experimental results demonstrate the effectiveness of our approach in elevating the performance of student models while diminishing computational demands, thereby enabling efficient, real-time medical image segmentation on resource-constrained devices.

FFYOLO: A Lightweight Forest Fire Detection Model Based on YOLOv8

Forest is an important resource for human survival, and forest fires are a serious threat to forest protection. Therefore, the early detection of fire and smoke is particularly important. Based on the manually set feature extraction method, the detection accuracy of the machine learning forest fire detection method is limited, and it is unable to deal with complex scenes. Meanwhile, most deep learning methods are difficult to deploy due to high computational costs. To address these issues, this paper proposes a lightweight forest fire detection model based on YOLOv8 (FFYOLO). Firstly, in order to better extract the features of fire and smoke, a channel prior dilatation attention module (CPDA) is proposed. Secondly, the mixed-classification detection head (MCDH), a new detection head, is designed. Furthermore, MPDIoU is introduced to enhance the regression and classification accuracy of the model. Then, in the Neck section, a lightweight GSConv module is applied to reduce parameters while maintaining model accuracy. Finally, the knowledge distillation strategy is used during training stage to enhance the generalization ability of the model and reduce the false detection. Experimental outcomes demonstrate that, in comparison to the original model, FFYOLO realizes an mAP0.5 of 88.8% on a custom forest fire dataset, which is 3.4% better than the original model, with 25.3% lower parameters and 9.3% higher frames per second (FPS).

Enhancing Few-Shot Learning in Lightweight Models via Dual-Faceted Knowledge Distillation.

In recent computer vision research, the pursuit of improved classification performance often leads to the adoption of complex, large-scale models. However, the actual deployment of such extensive models poses significant challenges in environments constrained by limited computing power and storage capacity. Consequently, this study is dedicated to addressing these challenges by focusing on innovative methods that enhance the classification performance of lightweight models. We propose a novel method to compress the knowledge learned by a large model into a lightweight one so that the latter can also achieve good performance in few-shot classification tasks. Specifically, we propose a dual-faceted knowledge distillation strategy that combines output-based and intermediate feature-based methods. The output-based method concentrates on distilling knowledge related to base class labels, while the intermediate feature-based approach, augmented by feature error distribution calibration, tackles the potential non-Gaussian nature of feature deviations, thereby boosting the effectiveness of knowledge transfer. Experiments conducted on MiniImageNet, CIFAR-FS, and CUB datasets demonstrate the superior performance of our method over state-of-the-art lightweight models, particularly in five-way one-shot and five-way five-shot tasks.

A new feature boosting based continual learning method for bearing fault diagnosis with incremental fault types

Rotating machinery can unexpectedly generate many new fault types under changing operating conditions. The capability of fault diagnostic models to adapt and acquire knowledge regarding new fault types is increasingly vital. However, traditional deep learning-based fault diagnosis models often encounter the challenge of catastrophic forgetting when facing new fault types. Numerous attempts were trapped with the stability–plasticity dilemma when tackling this phenomenon. In this study, we draw inspiration from the gradient boosting algorithm and propose a feature boosting based continual learning method. This method allows the diagnostic model to continuously and adaptively acquire knowledge of new fault types. Initially, the concept of gradient boosting is employed to construct an initial fault diagnostic model. Then, new modules are continuously extended dynamically for the initial diagnostic model to fit the residuals between the actual label and the output of the initial diagnostic model. Finally, to maintain the backbone of the fault diagnostic model as a single one, redundant parameters and feature dimensions are removed using an effective distillation strategy. Experimental results demonstrate that the feature boosting based continual learning method effectively mitigates catastrophic forgetting and enhances the plasticity of the fault diagnosis model, outperforming other existing methods.

Heterogeneous sampled subgraph neural networks with knowledge distillation to enhance double-blind compound-protein interaction prediction

Identifying binding compounds against a target protein is crucial for large-scale virtual screening in drug development. Recently, network-based methods have been developed for compound-protein interaction (CPI) prediction. However, they are difficult to be applied to unseen (i.e., never-seen-before) proteins and compounds. In this study, we propose SgCPI to incorporate local known interacting networks to predict CPI interactions. SgCPI randomly samples the local CPI network of the query compound-protein pair as a subgraph and applies a heterogeneous graph neural network (HGNN) to embed the active/inactive message of the subgraph. For unseen compounds and proteins, SgCPI-KD takes SgCPI as the teacher model to distillate its knowledge by estimating the potential neighbors. Experimental results indicate: (1) the sampled subgraphs of the CPI network introduce efficient knowledge for unseen molecular prediction with the HGNNs, and (2) the knowledge distillation strategy is beneficial to the double-blind interaction prediction by estimating molecular neighbors and distilling knowledge.

EPANet-KD: Efficient progressive attention network for fine-grained provincial village classification via knowledge distillation.

Fine-grained classification of historical traditional villages plays a crucial role in guiding the future development and construction of urban and rural areas. This study aims to propose a new dataset for fine-grained classification of traditional villages and to propose an efficient progressive attention network for the problem of low accuracy and efficiency of fine-grained traditional historical village classification. Firstly, in order to further study the long-standing problem of fine-grained classification of traditional villages, a new fine-grained classification dataset of traditional villages containing 4,400 images, referred to as PVCD, is proposed by crawling and hand-arranging. Secondly, a new Progressive Attention Module, abbreviated as PAM, is also proposed. PAM engages in attentional modeling of prominent spatial features within the spatial dimension, subsequently applying attentional modeling to channel features beneath the identified salient spatial features. This process involves salient spatial feature attention modeling of prominent channel features within the dimension to extract discriminative information for fine-grained classification, thereby enhancing the performance of classifying traditional villages with precision. Finally, a new knowledge distillation strategy of softened alignment distillation, or SAD for short, is proposed, which simply and efficiently transfers the knowledge of softened category probability distributions through. Notably, based on the above proposed PAM, the lightweight EPANet-Student and the heavyweight EPANet-Teacher are proposed. In addition, the heavyweight EPANet-Teacher transfers the knowledge of fine-grained categorization of traditional villages to the lightweight EPANet-Student through the proposed SAD, abbreviated as EPANet-KD. The experimental results show that the proposed EPANet-Teacher achieves state-of-the-art performance with an accuracy of 67.27%, and the proposed EPANet-KD achieves comparable performance to the proposed EPANet-Teacher with 3.32M parameters and 0.42G computation. The proposed EPANet-KD maintains a good balance of accuracy and efficiency in the fine-grained classification of traditional villages, considerably promoting the research on the fine-grained classification of traditional villages. In addition, it facilitates the digital preservation and development of traditional villages. All datasets, codes and benchmarking results are publicly available for the promotion of this research area. https://github.com/Jack13026212687/EPANet-KD.

Light-M: An efficient lightweight medical image segmentation framework for resource-constrained IoMT

The Internet of Medical Things (IoMT) is being incorporated into current healthcare systems. This technology intends to connect patients, IoMT devices, and hospitals over mobile networks, allowing for more secure, quick, and convenient health monitoring and intelligent healthcare services. However, existing intelligent healthcare applications typically rely on large-scale AI models, and standard IoMT devices have significant resource constraints. To alleviate this paradox, in this paper, we propose a Knowledge Distillation (KD)-based IoMT end-edge-cloud orchestrated architecture for medical image segmentation tasks, called Light-M, aiming to deploy a lightweight medical model in resource-constrained IoMT devices. Specifically, Light-M trains a large teacher model in the cloud server and employs computation in local nodes through imitation of the performance of the teacher model using knowledge distillation. Light-M contains two KD strategies: (1) active exploration and passive transfer (AEPT) and (2) self-attention-based inter-class feature variation (AIFV) distillation for the medical image segmentation task. The AEPT encourages the student model to learn undiscovered knowledge/features of the teacher model without additional feature layers, aiming to explore new features and outperform the teacher. To improve the distinguishability of the student for different classes, the student learns the self-attention-based feature variation (AIFV) between classes. Since the proposed AEPT and AIFV only appear in the training process, our framework does not involve any additional computation burden for a student model during the segmentation task deployment. Extensive experiments on cardiac images and public real-scene datasets demonstrate that our approach improves student model learning representations and outperforms state-of-the-art methods by combining two knowledge distillation strategies. Moreover, when deployed on the IoT device, the distilled student model takes only 29.6 ms for one sample at the inference step.

A novel Multi-Level Refined (MLR) knowledge graph design and chatbot system for healthcare applications.

Imagine having a knowledge graph that can extract medical health knowledge related to patient diagnosis solutions and treatments from thousands of research papers, distilled using machine learning techniques in healthcare applications. Medical doctors can quickly determine treatments and medications for urgent patients, while researchers can discover innovative treatments for existing and unknown diseases. This would be incredible! Our approach serves as an all-in-one solution, enabling users to employ a unified design methodology for creating their own knowledge graphs. Our rigorous validation process involves multiple stages of refinement, ensuring that the resulting answers are of the utmost professionalism and solidity, surpassing the capabilities of other solutions. However, building a high-quality knowledge graph from scratch, with complete triplets consisting of subject entities, relations, and object entities, is a complex and important task that requires a systematic approach. To address this, we have developed a comprehensive design flow for knowledge graph development and a high-quality entities database. We also developed knowledge distillation schemes that allow you to input a keyword (entity) and display all related entities and relations. Our proprietary methodology, multiple levels refinement (MLR), is a novel approach to constructing knowledge graphs and refining entities level-by-level. This ensures the generation of high-quality triplets and a readable knowledge graph through keyword searching. We have generated multiple knowledge graphs and developed a scheme to find the corresponding inputs and outputs of entity linking. Entities with multiple inputs and outputs are referred to as joints, and we have created a joint-version knowledge graph based on this. Additionally, we developed an interactive knowledge graph, providing a user-friendly environment for medical professionals to explore entities related to existing or unknown treatments/diseases. Finally, we have advanced knowledge distillation techniques.