Deep Transfer Learning Method Research Articles

Scattering moment matching-based interpretable domain adaptation for transfer diagnostic tasks

One significant fact familiar to the signal processing-based diagnostic community but generally ignored by the transfer learning-based diagnostic community is that the cyclostationarity of the monitored signal conveys the actual diagnostic information. Popular network architectures, e.g., ResNet, and domain discrepancy metrics, e.g., maximum mean discrepancy, in current transfer diagnostic research are generally borrowed from the transfer learning community yet do not explicitly consider machine fault physics. As Jérôme Antoni points out, signal processing and machine learning methods are not mutually exclusive but should complement each other. The current article aims to develop an interpretable domain adaptation method for transfer diagnostic tasks by simultaneously exploiting the ideas of cyclostationary signal processing and domain adaptation techniques. By taking NTScatNet as the network backbone and scattering moment distance as the domain discrepancy metric, the proposed scattering moment matching-based domain adaptation method is more interpretable and matches fault physics better than conventional deep transfer learning methods. Besides, the proposed method does not require target domain data during the training phase, thus relaxing the assumption of standard domain adaptation. The effectiveness and superiority of the proposed domain adaptation method were verified on four transfer diagnostic case studies, i.e., transfer diagnostic across bearing specifications, transfer diagnostic across escalator roller specifications, transfer diagnostic across transducers on bearing datasets, and transfer diagnostic across transducers on the gearbox datasets.

Bearing fault diagnosis based on transfer learning with dual-flow manifold ResNet and improved CapsNet

In intelligent fault diagnosis for rolling bearings, the variable operating conditions of rotating machinery lead to inconsistent distributions between training and testing data, inevitably decreasing fault diagnosis accuracy. To overcome this issue, we propose a bearing fault diagnosis method based on transfer learning with a dual-flow manifold residual network and improved capsule network (DMRCN). Firstly, we introduce a dual-flow manifold residual network structure consisting of continuous wide-kernel convolution, continuous narrow-kernel convolution, and manifold learning spatial information description blocks. This architecture is utilized to extract low-level features from the data. Then, we present a novel architecture of the capsule network with self-attention output. We incorporate a self-attention mechanism into the dynamic routing algorithm to transfer richer high-level features from the low-level capsule layer to the higher-level capsule layer. During training, we propose a loss function known as global-local distribution discrepancy loss to enhance the model’s generalization ability and accelerate model convergence. Finally, we validate the performance of DMRCN in cross-condition transfer fault diagnosis on both the Case Western Reserve University public dataset and a laboratory-built bearing experimental rig dataset. Experimental results demonstrate that DMRCN outperforms other commonly used deep transfer learning methods regarding fault diagnosis across different operating conditions.

Artificial intelligence and feature based transfer learning framework for diagnosis of chest diseases using radiology images

The subject of artificial intelligence-assisted diagnosis and design in the medical industry is very exciting due to considerable developments in medical imaging. In real-world applications, previous manual feature extraction strategies were inefficient in achieving the required results. The number of medical image databases is quickly increasing to accommodate hospital-based diseases as a result of the numerous uses of medical images in healthcare facilities, pathology, and medical diagnostic fields. The primary objective of this study is to create a computerized Artificial intelligence system that can accurately diagnose different diseases and reduce mistakes in the testing process. The study has two primary aspects. In the initial phase, we utilized the deep transfer learning method to extract the pertinent and crucial features from the image x-ray. Subsequently, the support vector machine employs these crucial extracted features to diagnose diseases from the x-ray14 dataset. The imbalanced dataset problem was also addressed with the utilization of the Synthetic Minority Oversampling Technique (SMOTE). The authors conduct a comparative analysis of the findings from this study in relation to other cutting-edge studies and employ cross-dataset experiments to evaluate its efficacy. The results demonstrate that the proposed approach has a detection accuracy of 95.2% for the disease. The VGG-16 model achieved 78.4% accuracy and an AUC of 90%. The proposed model can be applied to other diseases for further experiments.

A novel adaptive deep transfer learning method towards thermal error modeling of electric spindles under variable conditions

Thermal error modeling (TEM) plays a vital role in maintaining the machining accuracy of electric spindles. Recently, deep learning (DL) techniques have obtained promising achievements in this area. However, DL techniques have certain limitations. The data acquired from variable working conditions present large distribution discrepancies, the DL-based model established on one working condition fails to obtain satisfactory prediction accuracy in another working condition. Moreover, existing studies focus solely on using temperature features to build prediction models, neglecting the full use of multi-sensory information. To address these issues, this paper proposes a novel adaptive deep transfer learning method towards TEM of electric spindles, which takes full advantage of the temperature, current, and power sensory information. Firstly, finite element simulation is employed to analyze the thermal characteristics of the electric spindle and determine the locations for temperature measurement points. Then, the convolutional long short-term memory network (C-LSTMN) is constructed, where the spatial features from multi-sensory information between inputs and prediction patterns are captured by convolutional layers, these features are further processed by long short-term memory network (LSTMN) to extract temporal features. Subsequently, the multi-kernel joint maximum mean discrepancy (MK-JMMD) measure is developed to minimize the distribution discrepancies between the source and target domains, thus the prediction model initially established on the source domain can be adaptive to effectively predict on the target domain. Finally, with unavailable thermal error data in the target domain, the proposed method is validated through 12 transfer tasks using datasets from four working conditions with other comparison methods. The results demonstrate that the proposed method overcomes the challenges of unavailable labeled thermal error samples and outperforms advanced methods.

Multi-label deep transfer learning method for coupling fault diagnosis

With the development of complexity and integration of machines, the multiple components of the system are prone to simultaneous failures and the multi-fault signals may be measured in limited sensors. This fault mode is defined as coupling fault in this paper and becomes more common and attracts great attention. Current coupling fault diagnosis approaches rely on labeled data under consistent working conditions. However, in practical varying working conditions, the data is difficult to label and the traditional methods perform poorly. To address the challenge problem, we utilize some related annotated data under other working conditions for auxiliary learning and abstract this problem as a multi-label transfer learning problem which is a new research topic in fault diagnosis. We proposed a novel deep transfer coupling fault diagnosis method from the global and local views to achieve the effect of transfer, which is the first attempt to address this practical problem as far as we know. At the global level, we used Maximum Mean Discrepancy (MMD) at multiple stages of the network to reduce the hypothesis space of the model. Furthermore, we design a special manifold framework with multi-level similarity to constrain the hierarchy of the distance of the sample and the hierarchy of the labels being consistent between the two domains, to further improve the effect of local alignment. The proposed method has high accuracy in multiple transfer tasks for both public and laboratory datasets, powerfully demonstrating its effectiveness.

A new approach for heart disease detection using Motif transform-based CWT's time-frequency images with DenseNet deep transfer learning methods.

Electrocardiogram (ECG) signals are extensively utilized in the identification and assessment of diverse cardiac conditions, including congestive heart failure (CHF) and cardiac arrhythmias (ARR), which present potential hazards to human health. With the aim of facilitating disease diagnosis and assessment, advanced computer-aided systems are being developed to analyze ECG signals. This study proposes a state-of-the-art ECG data pattern recognition algorithm based on Continuous Wavelet Transform (CWT) as a novel signal preprocessing model. The Motif Transformation (MT) method was devised to diminish the drawbacks and limitations inherent in the CWT, such as the issue of boundary effects, limited localization in time and frequency, and overfitting conditions. This transformation technique facilitates the formation of diverse patterns (motifs) within the signals. The patterns (motifs) are constructed by comparing the amplitudes of each individual sample value in the ECG signals in terms of their largeness and smallness. In the subsequent stage, the obtained one-dimensional signals from the MT transformation were subjected to CWT to obtain scalogram images. In the last stage, the obtained scalogram images were subjected to classification using DenseNET deep transfer learning techniques. The combined approach of MT+CWT+DenseNET yielded an impressive success rate of 99.31 %.

A novel algorithm for complex transfer conditions in bearing fault diagnosis

Abstract Transfer learning in bearing fault diagnosis can effectively improve model generalization and accelerate the practical application of fault diagnosis algorithms. However, previous algorithms primarily focused on simple transfer conditions like known target domain data or the same device. In industrial practice, the conditions for algorithm transfer are more complex. Therefore, cross-domain fault diagnosis under complex transfer conditions is a challenging task with significant practical value. This paper proposes a new bearing fault diagnosis algorithm based on attention mechanism and feature enhancement, which provides better feature extraction capabilities. The main approach involves performing deep aliasing on deep features and training the model to identify domain-invariant classification features under extreme conditions for effective fault diagnosis. Additionally, our network performs well in handling low signal-to-noise ratio problems. Extensive experiments were conducted on three different bearing case studies to validate the effectiveness of the proposed method, showing superior performance compared to other deep transfer learning methods.

Electronic Nose Drift Suppression Based on Smooth Conditional Domain Adversarial Networks.

Anti-drift is a new and serious challenge in the field related to gas sensors. Gas sensor drift causes the probability distribution of the measured data to be inconsistent with the probability distribution of the calibrated data, which leads to the failure of the original classification algorithm. In order to make the probability distributions of the drifted data and the regular data consistent, we introduce the Conditional Adversarial Domain Adaptation Network (CDAN)+ Sharpness Aware Minimization (SAM) optimizer-a state-of-the-art deep transfer learning method.The core approach involves the construction of feature extractors and domain discriminators designed to extract shared features from both drift and clean data. These extracted features are subsequently input into a classifier, thereby amplifying the overall model's generalization capabilities. The method boasts three key advantages: (1) Implementation of semi-supervised learning, thereby negating the necessity for labels on drift data. (2) Unlike conventional deep transfer learning methods such as the Domain-adversarial Neural Network (DANN) and Wasserstein Domain-adversarial Neural Network (WDANN), it accommodates inter-class correlations. (3) It exhibits enhanced ease of training and convergence compared to traditional deep transfer learning networks. Through rigorous experimentation on two publicly available datasets, we substantiate the efficiency and effectiveness of our proposed anti-drift methodology when juxtaposed with state-of-the-art techniques.

Data stream classification using a deep transfer learning method based on extreme learning machine and recurrent neural network

Data stream classification using a deep transfer learning method based on extreme learning machine and recurrent neural network

SFR Modeling for Hybrid Power Systems Based on Deep Transfer Learning

A deep transfer learning method is presented for establishing the aggregated system frequency response (SFR) model of wind-thermal hybrid power systems (HPSs). In order to deal with nonlinearities and non-Gaussian disturbances, the quadratic survival information potential (QSIP) of the squared identification error is employed to construct the performance index when training recurrent neural networks (RNNs). A pre-trained SFR model is then obtained by the improved RNNs using the source domain data collected from the HPS in historical scenarios. Subsequently, the maximum mean difference is utilized to test the similarity of the HPS in historical and current scenarios. After that, the pre-trained SFR model is fine-tuned by adding some nodes to the recurrent layer and a functional link to the input layer. The SFR model of the HPS operating in current scenario can then be built based on the transferred source domain pre-trained SFR model. Simulation results illustrate that the proposed data driven modelling method can obtain accurate, effective and timely SFR model for a wind-thermal HPS with different wind speeds and load disturbances.

Unsupervised Deep Transfer Learning Method for Rolling Bearing Fault Diagnosis Based on Improved Convolutional Neural Network

As a critical component widely used in mechanical equipment, the implementation of intelligent fault diagnosis for rolling bearings can improve the reliability of equipment. In this paper, a method named JMMD-CKDSCNet is proposed to address the task of fault diagnosis under unsupervised domain discrepancy scenarios. First, the convolutional kernel dropout (CKD) mechanism is introduced in the convolutional layer, and partial convolutional kernel weights are set to be inactive during the training process using the random mask. Second, skip connection (SC) fuses the features of multiple shallow layers to preserve and transfer the original features. Finally, domain alignment is achieved using joint maximum mean discrepancy (JMMD), which measures the joint distribution between different domains with feature discrepancies under the condition that the target domain lacks labeled data. The experimental results demonstrate that CKDSCNet exhibits superior generalization performance and outperforms other models in terms of diagnostic accuracy and model performance. Compared with other domain adaptation methods, JMMD has significant superiority, proving the application value of JMMD-CKDSCNet.

An Improved Deep Transfer Learning Method for Rotating Machinery Fault Diagnosis Based on Time Frequency Diagram and Pretraining Model

An Improved Deep Transfer Learning Method for Rotating Machinery Fault Diagnosis Based on Time Frequency Diagram and Pretraining Model

A Novel Unsupervised Deep Transfer Learning Method With Isolation Forest for Machine Fault Diagnosis

Recent diagnostic approaches based on the deep learning model have attracted much attention. However, developing an outstanding AI diagnostic model requires many training samples with labeled information. Moreover, training deep models is labor-intensive and time-consuming, and labeling samples and training models increase workload. To overcome these problems, this paper proposes an unsupervised deep transfer learning method with isolation forest for machine fault diagnosis. First, the isolation forest is used to classify and label the samples automatically; then, these labeled data are used to train deep learning models; finally, small data with the label of the target domain are used to fine-tune parameters and complete the fault diagnosis. The proposed approach has been validated with the fan gearbox dataset, the bearing dataset, and the ball screw dataset. The results show that the proposed unsupervised deep transfer learning model has high accuracy and generality.

TrGPCR:GPCR-ligand Binding Affinity Predicting based on Dynamic Deep Transfer Learning.

Predicting G protein-coupled receptor (GPCR)-ligand binding affinity plays a crucial role in drug development. However, determining GPCR-ligand binding affinities is time-consuming and resource-intensive. Although many studies used data-driven methods to predict binding affinity, most of these methods required protein 3D structure, which was often unknown. Moreover, part of these studies only considered the sequence characteristics of the protein, ignoring the secondary structure of the protein. The number of known GPCR for affinity prediction is only a few thousand, which is insufficient for deep learning training. Therefore, this study aimed to propose a deep transfer learning method called TrGPCR, which used dynamic transfer learning to solve the problem of insufficient GPCR data. We used the Binding Database(BindingDB) as the source domain and the GLASS(GPCR-Ligand Association) database as the target domain. We also introduced protein secondary structures, called pockets, as features to predict binding affinities. Compared with DeepDTA, our model improved by 5.2% on RMSE(root mean square error) and 4.5% on MAE(mean squared error).

Discrimination of benign and malignant breast lesions on dynamic contrast-enhanced magnetic resonance imaging using deep learning.

To evaluate the capability of deep transfer learning (DTL) and fine-tuning methods in differentiating malignant from benign lesions in breast dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI). The diagnostic efficiencies of the VGG19, ResNet50, and DenseNet201 models were tested under the same dataset. The model with the highest performance was selected and modified utilizing three fine-tuning strategies (S1-3). Fifty additional lesions were selected to form the validation set to verify the generalization abilities of these models. The accuracy (Ac) of the different models in the training and test sets, as well as the precision (Pr), recall rate (Rc), F1 score (), and area under the receiver operating characteristic curve (AUC), were primary performance indicators. Finally, the kappa test was used to compare the degree of agreement between the DTL models and pathological diagnosis in differentiating malignant from benign breast lesions. The Pr, Rc, f1, and AUC of VGG19 (86.0%, 0.81, 0.81, and 0.81, respectively) were higher than those of DenseNet201 (70.0%, 0.61, 0.63, and 0.61, respectively) and ResNet50 (61.0%, 0.59, 0.59, and 0.59). After fine-tuning, the Pr, Rc, f1, and AUC of S1 (87.0%, 0.86, 0.86, and 0.86, respectively) were higher than those of VGG19. Notably, the degree of agreement between S1 and pathological diagnosis in differentiating malignant from benign breast lesions was 0.720 (κ = 0.720), which was higher than that of DenseNet201 (κ = 0.440), VGG19 (κ = 0.640), and ResNet50 (κ = 0.280). The VGG19 model is an effective method for identifying benign and malignant breast lesions on DCE-MRI, and its performance can be further improved via fine-tuning. Overall, our findings insinuate that this technique holds potential clinical application value.

Addressing data scarcity in protein fitness landscape analysis: A study on semi-supervised and deep transfer learning techniques

This paper presents a comprehensive analysis of deep transfer learning methods, supervised methods, and semi-supervised methods in the context of protein fitness prediction, with a focus on small datasets. The analysis includes the exploration of the combination of different data sources to enhance the performance of the models. While deep learning and deep transfer learning methods have shown remarkable performance in situations with abundant data, this study aims to address the more realistic scenario faced by wet lab researchers, where labeled data is often limited.The novelty of this work lies in its examination of deep transfer learning in the context of small datasets and its consideration of semi-supervised methods and multi-view strategies. While previous research has extensively explored deep transfer learning in large dataset scenarios, little attention has been given to its efficacy in small dataset settings or its comparison with semi-supervised approaches.Our findings suggest that deep transfer learning, exemplified by ProteinBERT, shows promising performance in this context compared to the rest of the methods across various evaluation metrics, not only in small dataset contexts but also in large dataset scenarios. This highlights the robustness and versatility of deep transfer learning in protein fitness prediction tasks, even with limited labeled data.The results of this study shed light on the potential of deep transfer learning as a state-of-the-art approach in the field of protein fitness prediction. By leveraging pre-trained models and fine-tuning them on small datasets, researchers can achieve competitive performance surpassing traditional supervised and semi-supervised methods. These findings provide valuable insights for wet lab researchers who face the challenge of limited labeled data, enabling them to make informed decisions when selecting the most effective methodology for their specific protein fitness prediction tasks.Additionally, the study investigated the combination of two different sources of information (encodings) through our enhanced semi-supervised methods, yielding noteworthy results improving their base model and providing valuable insights for further research.The presented analysis contributes to a better understanding of the capabilities and limitations of different learning approaches in small dataset scenarios, ultimately aiding in the development of improved protein fitness prediction methods.

A White Dwarf Search Model Based on a Deep Transfer-learning Method

White dwarfs represent the ultimate stage of evolution for over 97% of stars and play a crucial role in studies of the Milky Way’s structure and evolution. Recent years have witnessed significant progress in using deep-learning methods for identifying unique objects in large-scale data. In this paper, we present a model based on transfer learning for identifying white dwarfs. We constructed a data set using the spectra released by LAMOST DR9 and trained a convolutional neural network model. The model was then further trained using a transfer-learning approach for a binary classification model. Our final model is comprised of a seven-class classification model and a binary classification model. The testing set yielded an accuracy rate of 96.08%. Our proposed model successfully identifies 4314 of the 4479 white dwarfs published in previous papers. We applied this model to filter the 1,121,128 spectral data from the LAMOST DR9 V1 catalog. Subsequently, we obtained 6317 white dwarf candidates, of which 5014 were cross-validated and found to be known white dwarfs. We finally identified 489 new white dwarfs out of the remaining 1303 candidates, containing 377 DAs, 1 DB, 4 DZs, 1 magnetic WD, 101 DA+M binaries, and 1 DB+M binary. Our study also compared transfer-learning methods with non-transfer-learning methods, and the results show that transfer learning provides faster training speed and a higher accuracy rate. We provide the trained model and a corresponding usage program for subsequent studies.

A face image classification method of autistic children based on the two-phase transfer learning.

Autism spectrum disorder (ASD) is a neurodevelopmental disorder, which seriously affects children's normal life. Screening potential autistic children before professional diagnose is helpful to early detection and early intervention. Autistic children have some different facial features from non-autistic children, so the potential autistic children can be screened by taking children's facial images and analyzing them with a mobile phone. The area under curve (AUC) is a more robust metrics than accuracy in evaluating the performance of a model used to carry out the two-category classification, and the AUC of the deep learning model suitable for the mobile terminal in the existing research can be further improved. Moreover, the size of an input image is large, which is not fit for a mobile phone. A deep transfer learning method is proposed in this research, which can use images with smaller size and improve the AUC of existing studies. The proposed transfer method uses the two-phase transfer learning mode and the multi-classifier integration mode. For MobileNetV2 and MobileNetV3-Large that are suitable for a mobile phone, the two-phase transfer learning mode is used to improve their classification performance, and then the multi-classifier integration mode is used to integrate them to further improve the classification performance. A multi-classifier integrating calculation method is also proposed to calculate the final classification results according to the classifying results of the participating models. The experimental results show that compared with the one-phase transfer learning, the two-phase transfer learning can significantly improve the classification performance of MobileNetV2 and MobileNetV3-Large, and the classification performance of the integrated classifier is better than that of any participating classifiers. The accuracy of the integrated classifier in this research is 90.5%, and the AUC is 96.32%, which is 3.51% greater than the AUC (92.81%) of the previous studies.

Classification of Unmanned Aerial Vehicle and Bird Images Using Deep Transfer Learning Methods

The increasing accessibility and affordability of unmanned aerial vehicles (UAVs), commonly known as drones, have led to the emergence of malicious users. In precaution to this perceived threat, various anti-UAV systems are being developed, including electro-optical systems utilizing cameras. It is possible to detect UAVs from images using various machine learning methods. However, the similarity between UAVs and birds poses a challenge, as birds can be mistakenly identified as UAVs, leading to false alarms in a security system. In order to avoid this problem, this study provided the classification of birds and unmanned aerial vehicles over images using deep learning methods. In this study, a data set consisting of 400 birds and 428 UAV images was used. The data were divided into 70% for training, 30% for testing and validation purposes. Three different deep learning models, based on DenseNet, VGG16, and VGG19 architectures, were trained using transfer learning techniques, and their performances were compared. Experimental results on the test data showed an accuracy of 94.64% with the DenseNet model, 89.67% with the VGG16 model, and 90.67% with the VGG19 model.

Deep transfer learning for inter-chain contact predictions of transmembrane protein complexes

Membrane proteins are encoded by approximately a quarter of human genes. Inter-chain residue-residue contact information is important for structure prediction of membrane protein complexes and valuable for understanding their molecular mechanism. Although many deep learning methods have been proposed to predict the intra-protein contacts or helix-helix interactions in membrane proteins, it is still challenging to accurately predict their inter-chain contacts due to the limited number of transmembrane proteins. Addressing the challenge, here we develop a deep transfer learning method for predicting inter-chain contacts of transmembrane protein complexes, named DeepTMP, by taking advantage of the knowledge pre-trained from a large data set of non-transmembrane proteins. DeepTMP utilizes a geometric triangle-aware module to capture the correct inter-chain interaction from the coevolution information generated by protein language models. DeepTMP is extensively evaluated on a test set of 52 self-associated transmembrane protein complexes, and compared with state-of-the-art methods including DeepHomo2.0, CDPred, GLINTER, DeepHomo, and DNCON2_Inter. It is shown that DeepTMP considerably improves the precision of inter-chain contact prediction and outperforms the existing approaches in both accuracy and robustness.