Knowledge Transfer Learning Research Articles

MT-Net: Single image dehazing based on meta learning, knowledge transfer and contrastive learning

Single image dehazing is becoming increasingly important as its results impact the efficiency of subsequent computer vision tasks. While many methods have been proposed to address this challenge, existing dehazing approaches often exhibit limited adaptability to different types of images and lack future learnability. In light of this, we propose a dehazing network based on meta-learning, knowledge transfer, and contrastive learning, abbreviated as MT-Net. In our approach, we combine knowledge transfer with meta-learning to tackle these challenges, thus enhancing the network’s generalization performance. We refine the structure of knowledge transfer by introducing a two-phases approach to facilitate learning under the guidance of teacher networks and learning committee networks. We also optimize the negative examples of contrastive learning to reduce the contrast space. Extensive experiments conducted on synthetic and real datasets demonstrate the remarkable performance of our method in both quantitative and qualitative comparisons. The code has been released on https://github.com/71717171fan/MT-Net.

Enhancing predictions of experimental band gap using machine learning and knowledge transfer

Density functional theory (DFT) calculations using the generalized gradient approximation (GGA) functional are known to underestimate the bandgap (Eg) compared with experimental values. Although hybrid functionals can address this issue, they are limited by their high computational costs. This paper presents a novel and efficient method to predict experimental Eg more accurately using machine learning (ML) models enhanced with knowledge transfer learning. We developed surrogate models based on the Eg calculated using the GGA functional and easily obtainable features from chemical formulas. These models exhibited superior prediction accuracy, with a coefficient of determination (R²) of 0.817 and a mean absolute error (MAE) of 0.289 eV. We further performed feature importance analysis and symbolic regression to provide a detailed interpretation of the ML models. It was verified that various features, such as the potential to form ionic compounds and the number of elements in a material, have a major influence on experimental Eg predictions. Based on these results, an interpretable formula for experimental Eg was derived, which can be used in future studies. Consequently, we developed a platform for predicting the experimental Eg values of inorganic materials, effectively narrowing the gap between theoretical calculations and experiments, which can be used to screen novel materials with promising properties.

Dynamic collaborative learning with heterogeneous knowledge transfer for long-tailed visual recognition

Solving the long-tailed visual recognition with deep convolutional neural networks is still a challenging task. As a mainstream method, multi-experts models achieve SOTA accuracy for tackling this problem, but the uncertainty in network learning and the complexity in fusion inference constrain the performance and practicality of the multi-experts models. To remedy this, we propose a novel dynamic collaborative learning with heterogeneous knowledge transfer model (DCHKT) in this paper, in which experts with different expertise collaborate to make predictions. DCHKT consists of two core components: dynamic adaptive weight adjustment and heterogeneous knowledge transfer learning. First, the dynamic adaptive weight adjustment is designed to shift the focus of model training between the global expert and domain experts via dynamic adaptive weight. By modulating the trade-off between the learning of features and classifier, the dynamic adaptive weight adjustment can enhance the discriminative ability of each expert and alleviate the uncertainty of model learning. Then, heterogeneous knowledge transfer learning, which measures the distribution differences between the fusion logits of multiple experts and the predicted logits of each expert with different specialties, can achieve message passing between experts and enhance the consistency of ensemble prediction in model training and inference to promote their collaborations. Finally, extensive experimental results on public long-tailed datasets: CIFAR-LT, ImageNet-LT, Place-LT and iNaturalist2018, demonstrate the effectiveness and superiority of our DCHKT.

Combining graph neural networks and transformers for few-shot nuclear receptor binding activity prediction

Nuclear receptors (NRs) play a crucial role as biological targets in drug discovery. However, determining which compounds can act as endocrine disruptors and modulate the function of NRs with a reduced amount of candidate drugs is a challenging task. Moreover, the computational methods for NR-binding activity prediction mostly focus on a single receptor at a time, which may limit their effectiveness. Hence, the transfer of learned knowledge among multiple NRs can improve the performance of molecular predictors and lead to the development of more effective drugs. In this research, we integrate graph neural networks (GNNs) and Transformers to introduce a few-shot GNN-Transformer, Meta-GTNRP to predict the binding activity of compounds using the combined information of different NRs and identify potential NR-modulators with limited data. The Meta-GTNRP model captures the local information in graph-structured data and preserves the global-semantic structure of molecular graph embeddings for NR-binding activity prediction. Furthermore, a few-shot meta-learning approach is proposed to optimize model parameters for different NR-binding tasks and leverage the complementarity among multiple NR-specific tasks to predict binding activity of compounds for each NR with just a few labeled molecules. Experiments with a compound database containing annotations on the binding activity for 11 NRs shows that Meta-GTNRP outperforms other graph-based approaches. The data and code are available at: https://github.com/ltorres97/Meta-GTNRP.Scientific contributionThe proposed few-shot GNN-Transformer model, Meta-GTNRP captures the local structure of molecular graphs and preserves the global-semantic information of graph embeddings to predict the NR-binding activity of compounds with limited available data; A few-shot meta-learning framework adapts model parameters across NR-specific tasks for different NRs in a joint learning procedure to predict the binding activity of compounds for each NR with just a few labeled molecules in highly imbalanced data scenarios; Meta-GTNRP is a data-efficient approach that combines the strengths of GNNs and Transformers to predict the NR-binding properties of compounds through an optimized meta-learning procedure and deliver robust results valuable to identify potential NR-based drug candidates.

ORPP—An Ontology for Skill-Based Robotic Process Planning in Agile Manufacturing

Ontology plays a significant role in AI (Artificial Intelligence) and robotics by providing structured data, reasoning, action understanding, context awareness, knowledge transfer, and semantic learning. The structured framework created by the ontology for knowledge representation is crucial for enabling intelligent behavior in robots. This paper provides a state-of-the-art analysis on the existing ontology approaches and at the same time consolidates the terms in the robotic task planning domain. The major gap identified in the literature is the need to bridge higher-level robotic process management and lower-level robotic control. This gap makes it difficult for operators/non-robotic experts to integrate robots into their production processes as well as evaluate key performance indicators (KPI) of the processes. To fill the gap, the authors propose an ontology for skill-based robotics process planning (ORPP). ORPP not only provides a standardization in the robotic process planning in the agile manufacturing domain but also enables non-robotic experts to design and plan their production processes using an intuitive Process-Task-Skill-Primitive structure to control low-level robotic actions. On the performance level, this structure provides traceability of the KPIs down to the robot control level.

Domain adaptive noise reduction with iterative knowledge transfer and style generalization learning

Low-dose computed tomography (LDCT) denoising tasks face significant challenges in practical imaging scenarios. Supervised methods encounter difficulties in real-world scenarios as there are no paired data for training. Moreover, when applied to datasets with varying noise patterns, these methods may experience decreased performance owing to the domain gap. Conversely, unsupervised methods do not require paired data and can be directly trained on real-world data. However, they often exhibit inferior performance compared to supervised methods. To address this issue, it is necessary to leverage the strengths of these supervised and unsupervised methods. In this paper, we propose a novel domain adaptive noise reduction framework (DANRF), which integrates both knowledge transfer and style generalization learning to effectively tackle the domain gap problem. Specifically, an iterative knowledge transfer method with knowledge distillation is selected to train the target model using unlabeled target data and a pre-trained source model trained with paired simulation data. Meanwhile, we introduce the mean teacher mechanism to update the source model, enabling it to adapt to the target domain. Furthermore, an iterative style generalization learning process is also designed to enrich the style diversity of the training dataset. We evaluate the performance of our approach through experiments conducted on multi-source datasets. The results demonstrate the feasibility and effectiveness of our proposed DANRF model in multi-source LDCT image processing tasks. Given its hybrid nature, which combines the advantages of supervised and unsupervised learning, and its ability to bridge domain gaps, our approach is well-suited for improving practical low-dose CT imaging in clinical settings. Code for our proposed approach is publicly available at https://github.com/tyfeiii/DANRF.

Semantic Pre-Alignment and Ranking Learning With Unified Framework for Cross-Modal Retrieval

Cross-modal retrieval aims at retrieving highly semantic relevant information among multi-modalities. Existing cross-modal retrieval methods mainly explore the semantic consistency between image and text while rarely consider the rankings of positive instances in the retrieval results. Moreover, these methods seldom take into account the cross-interaction between image and text, which leads to the deficiency of learning their semantic relations. In this paper, we propose a Unified framework with Ranking Learning (URL) for cross-modal retrieval. The unified framework consists of three sub-networks, visual network, textual network, and interaction network. Visual network and textual network project the image feature and text feature into their corresponding hidden spaces respectively. Then, the interaction network forces the target image-text representation to align in the common space. For unifying both semantics and rankings, we propose a new optimization paradigm including pre-alignment for semantic knowledge transfer and ranking learning for final retrieval, which can decouple semantic alignment and ranking learning. The former focuses on the semantic pre-alignment optimized by semantic classification and the latter revolves around the retrieval rankings. For the ranking learning, we introduce a cross-AP loss which can directly optimize the retrieval metric average precision for cross-modal retrieval. We conduct experiments on four widely-used benchmarks, including Wikipedia dataset, Pascal Sentence dataset, NUS-WIDE-10k dataset, and PKU XMediaNet dataset respectively. Extensive experimental results show that the proposed method can obtain higher retrieval precision.

Masked Image Modeling Auxiliary Pseudo-Label Propagation with a Clustering Central Rectification Strategy for Cross-Scene Classification

Cross-scene classification focuses on setting up an effective domain adaptation (DA) way to transfer the learnable knowledge from source to target domain, which can be reasonably achieved through the pseudo-label propagation procedure. However, it is hard to bridge the objective existing severe domain discrepancy between source and target domains, and thus, there are several unreliable pseudo-labels generated in target domain and involved into pseudo-label propagation procedure, which would lead to unreliable error accumulation to deteriorate the performance of cross-scene classification. Therefore, in this paper, a novel Masked Image Modeling Auxiliary Pseudo-Label Propagation called MIM-AP2 with clustering central rectification strategy is proposed to improve the quality of pseudo-label propagation for cross-scene classification. First, in order to gracefully bridge the domain discrepancy and improve DA representation ability in-domain, a supervised class-token contrastive learning is designed to find the more consistent contextual clues to achieve knowledge transfer learning from source to target domain. At the same time, it is also incorporated with a self-supervised MIM mechanism according to a low random masking ratio to capture domain-specific information for improving the discriminability in-domain, which can lay a solid foundation for high-quality pseudo-label generation. Second, aiming to alleviate the impact of unreliable error accumulation, a clustering central rectification strategy is designed to adaptively update robustness clustering central representations to assist in rectifying unreliable pseudo-labels and learning a superior target domain specific classifier for cross-scene classification. Finally, extensive experiments are conducted on six cross-scene classification benchmarks, and the results are superior to other DA methods. The average accuracy reached 95.79%, which represents a 21.87% improvement over the baseline. This demonstrates that the proposed MIM-AP2 can provide significantly improved performance.

Transfer learning and self-distillation for automated detection of schizophrenia using single-channel EEG and scalogram images.

Schizophrenia (SZ) has been acknowledged as a highly intricate mental disorder for a long time. In fact, individuals with SZ experience a blurred line between fantasy and reality, leading to a lack of awareness about their condition, which can pose significant challenges during the treatment process. Due to the importance of the issue, timely diagnosis of this illness can not only assist patients and their families in managing the condition but also enable early intervention, which may help prevent its advancement. EEG is a widely utilized technique for investigating mental disorders like SZ due to its non-invasive nature, affordability, and wide accessibility. In this study, our main goal is to develop an optimized system that can achieve automatic diagnosis of SZ with minimal input information. To optimize the system, we adopted a strategy of using single-channel EEG signals and integrated knowledge distillation and transfer learning techniques into the model. This approach was designed to improve the performance and efficiency of our proposed method for SZ diagnosis. Additionally, to leverage the pre-trained models effectively, we converted the EEG signals into images using Continuous Wavelet Transform (CWT). This transformation allowed us to harness the capabilities of pre-trained models in the image domain, enabling automatic SZ detection with enhanced efficiency. To achieve a more robust estimate of the model's performance, we employed fivefold cross-validation. The accuracy achieved from the 5-s records of the EEG signal, along with the combination of self-distillation and VGG16 for the P4 channel, is 97.81. This indicates a high level of accuracy in diagnosing SZ using the proposed method.

Advances in cancer DNA methylation analysis with methPLIER: use of non-negative matrix factorization and knowledge-based constraints to enhance biological interpretability

DNA methylation is an epigenetic modification that results in dynamic changes during ontogenesis and cell differentiation. DNA methylation patterns regulate gene expression and have been widely researched. While tools for DNA methylation analysis have been developed, most of them have focused on intergroup comparative analysis within a dataset; therefore, it is difficult to conduct cross-dataset studies, such as rare disease studies or cross-institutional studies. This study describes a novel method for DNA methylation analysis, namely, methPLIER, which enables interdataset comparative analyses. methPLIER combines Pathway Level Information Extractor (PLIER), which is a non-negative matrix factorization (NMF) method, with regularization by a knowledge matrix and transfer learning. methPLIER can be used to perform intersample and interdataset comparative analysis based on latent feature matrices, which are obtained via matrix factorization of large-scale data, and factor-loading matrices, which are obtained through matrix factorization of the data to be analyzed. We used methPLIER to analyze a lung cancer dataset and confirmed that the data decomposition reflected sample characteristics for recurrence-free survival. Moreover, methPLIER can analyze data obtained via different preprocessing methods, thereby reducing distributional bias among datasets due to preprocessing. Furthermore, methPLIER can be employed for comparative analyses of methylation data obtained from different platforms, thereby reducing bias in data distribution due to platform differences. methPLIER is expected to facilitate cross-sectional DNA methylation data analysis and enhance DNA methylation data resources.

Knowledge Distillation and Transfer Learning Combined for Innovative Visualization Teaching of Non-Heritage Designs

Abstract This study introduces a novel approach by combining knowledge distillation and transfer learning to create a model that, despite its smaller size, approaches the accuracy of its much larger counterparts. It leverages a trained model from a source domain (teacher) to enhance an untrained model in a target domain (student) with significantly fewer parameters. Through transfer learning, we utilize pre-trained deep learning model parameters as initial values. This paper also explores integrating intangible cultural heritage (ICH) information with school curricula, transforming traditional knowledge presentation into intuitive, personalized displays. Our findings highlight that ICH visualization spans nine categories, with traditional arts and crafts leading with 25 items. Interestingly, only 22.58% of students understand Native American culture, pointing towards the potential for educational enhancement. The research suggests designing curriculum with varied teaching activities to improve students’ comprehensive skills.

Knowledge transfer and organizational learning in management practice in international automotive company

Knowledge transfer and organizational learning in management practice in international automotive company

Integrating Knowledge Distillation and Transfer Learning for Enhanced QoT-Estimation in Optical Networks

Integrating Knowledge Distillation and Transfer Learning for Enhanced QoT-Estimation in Optical Networks

A Fast Evolutionary Knowledge Transfer Search for Multiscale Deep Neural Architecture.

The emergence of neural architecture search (NAS) algorithms has removed the constraints on manually designed neural network architectures, so that neural network development no longer requires extensive professional knowledge, trial and error. However, the extremely high computational cost limits the development of NAS algorithms. In this article, in order to reduce computational costs and to improve the efficiency and effectiveness of evolutionary NAS (ENAS) is investigated. In this article, we present a fast ENAS framework for multiscale convolutional networks based on evolutionary knowledge transfer search (EKTS). This framework is novel, in that it combines global optimization methods with local optimization methods for search, and searches a multiscale network architecture. In this article, evolutionary computation is used as a global optimization algorithm with high robustness and wide applicability for searching neural architectures. At the same time, for fast search, we combine knowledge transfer and local fast learning to improve the search speed. In addition, we explore a multiscale gray-box structure. This gray box structure combines the Bandelet transform with convolution to improve network approximation, learning, and generalization. Finally, we compare the architectures with more than 40 different neural architectures, and the results confirmed its effectiveness.

A light-weight skeleton human action recognition model with knowledge distillation for edge intelligent surveillance applications

Skeleton based human action recognition has evolved as one of the most important applications in multimedia IoT system. However, it requires extensive computation resource including high performance computing unites and large memory to train a deep mode with large number of parameters, which seriously limits it effectiveness and efficiency for edge intelligence multimedia IoT applications. In this paper, a knowledge distillation based light-weight deep model is proposed for skeleton human action recognition to meet the edge multimedia IoT applications. It can get competitive recognition performance in terms of learning accuracy for combination of AI model and edge surveillance equipment. On the one hand, to achieve desirable accuracy, we propose a deep pose-transition image representation method based on two-stream spatial–temporal architecture, which can mine the hidden features of color texture images in spatial and temporal domain, and fuse them for comprehensive discrimination before final classification. On the other hand, to increase the transfer learning ability to the student model on the edge device, we use tucker decomposition to weak the teacher model during knowledge transfer learning process. Finally, in order to validate the effectiveness of our proposal, we conducted extensive experiments to evaluate the proposed approach. The experimental results demonstrate that our proposal can realize deep model miniaturization to meet the requirement of edge multimedia IoT system and achieve the competitive performance.

A fault diagnosis online method of railway track based on knowledge transfer learning

A fault diagnosis online method of railway track based on knowledge transfer learning

Knowledge Transfer Learning via Dual Density Sampling for Resource-Limited Domain Adaptation

Knowledge Transfer Learning via Dual Density Sampling for Resource-Limited Domain Adaptation

A Cross-Domain Recommender System for Literary Books Using Multi-Head Self-Attention Interaction and Knowledge Transfer Learning

Existing book recommendation methods often overlook the rich information contained in the comment text, which can limit their effectiveness. Therefore, a cross-domain recommender system for literary books that leverages multi-head self-attention interaction and knowledge transfer learning is proposed. Firstly, the BERT model is employed to obtain word vectors, and CNN is used to extract user and project features. Then, higher-level features are captured through the fusion of multi-head self-attention and addition pooling. Finally, knowledge transfer learning is introduced to conduct joint modeling between different domains by simultaneously extracting domain-specific features and shared features between domains. On the Amazon dataset, the proposed model achieved MAE and MSE of 0.801 and 1.058 in the “movie-book” recommendation task and 0.787 and 0.805 in the “music-book” recommendation task, respectively. This performance is significantly superior to other advanced recommendation models. Moreover, the proposed model also has good universality on the Chinese dataset.

Dual selections based knowledge transfer learning for cross-subject motor imagery EEG classification

IntroductionMotor imagery electroencephalograph (MI-EEG) has attracted great attention in constructing non-invasive brain-computer interfaces (BCIs) due to its low-cost and convenience. However, only a few MI-EEG classification methods have been recently been applied to BCIs, mainly because they suffered from sample variability across subjects. To address this issue, the cross-subject scenario based on domain adaptation has been widely investigated. However, existing methods often encounter problems such as redundant features and incorrect pseudo-label predictions in the target domain.MethodsTo achieve high performance cross-subject MI-EEG classification, this paper proposes a novel method called Dual Selections based Knowledge Transfer Learning (DS-KTL). DS-KTL selects both discriminative features from the source domain and corrects pseudo-labels from the target domain. The DS-KTL method applies centroid alignment to the samples initially, and then adopts Riemannian tangent space features for feature adaptation. During feature adaptation, dual selections are performed with regularizations, which enhance the classification performance during iterations.Results and discussionEmpirical studies conducted on two benchmark MI-EEG datasets demonstrate the feasibility and effectiveness of the proposed method under multi-source to single-target and single-source to single-target cross-subject strategies. The DS-KTL method achieves significant classification performance improvement with similar efficiency compared to state-of-the-art methods. Ablation studies are also conducted to evaluate the characteristics and parameters of the proposed DS-KTL method.

Phase retrieval for X-ray differential phase contrast radiography with knowledge transfer learning from virtual differential absorption model

Grating-based X-ray phase contrast radiography and computed tomography (CT) are promising modalities for future medical applications. However, the ill-posed phase retrieval problem in X-ray phase contrast imaging has hindered its use for quantitative analysis in biomedical imaging. Deep learning has been proved as an effective tool for image retrieval. However, in practical grating-based X-ray phase contrast imaging system, acquiring the ground truth of phase to form image pairs is challenging, which poses a great obstacle for using deep leaning methods. Transfer learning is widely used to address the problem with knowledge inheritance from similar tasks. In the present research, we propose a virtual differential absorption model and generate a training dataset with differential absorption images and absorption images. The knowledge learned from the training is transferred to phase retrieval with transfer learning techniques. Numerical simulations and experiments both demonstrate its feasibility. Image quality of retrieved phase radiograph and phase CT slices is improved when compared with representative phase retrieval methods. We conclude that this method is helpful in both X-ray 2D and 3D imaging and may find its applications in X-ray phase contrast radiography and X-ray phase CT.