Transfer Learning Strategy Research Articles

Transfer Learning-Based Specific Emitter Identification for ADS-B over Satellite System

In future aviation surveillance, the demand for higher real-time updates for global flights can be met by deploying automatic dependent surveillance–broadcast (ADS-B) receivers on low Earth orbit satellites, capitalizing on their global coverage and terrain-independent capabilities for seamless monitoring. Specific emitter identification (SEI) leverages the distinctive features of ADS-B data. High data collection and annotation costs, along with limited dataset size, can lead to overfitting during training and low model recognition accuracy. Transfer learning, which does not require source and target domain data to share the same distribution, significantly reduces the sensitivity of traditional models to data volume and distribution. It can also address issues related to the incompleteness and inadequacy of communication emitter datasets. This paper proposes a distributed sensor system based on transfer learning to address the specific emitter identification. Firstly, signal fingerprint features are extracted using a bispectrum transform (BST) to train a convolutional neural network (CNN) preliminarily. Decision fusion is employed to tackle the challenges of the distributed system. Subsequently, a transfer learning strategy is employed, incorporating frozen model parameters, maximum mean discrepancy (MMD), and classification error measures to reduce the disparity between the target and source domains. A hyperbolic space module is introduced before the output layer to enhance the expressive capacity and data information extraction. After iterative training, the transfer learning model is obtained. Simulation results confirm that this method enhances model generalization, addresses the issue of slow convergence, and leads to improved training accuracy.

ACVPred: Enhanced prediction of anti-coronavirus peptides by transfer learning combined with data augmentation

Anti-coronavirus peptides (ACVPs) have garnered significant attention in COVID-19 therapeutic research due to their precise targeting, low risk of drug resistance, flexible synthesis, and effectiveness against viral mutations. Although some in-silico methods have been developed to predict ACVPs, they suffer from challenges such as limited datasets and a lack of interpretability. Hence, this study introduces ACVPred, an algorithm for ACVP prediction, based on two few-shot learning strategies: transfer learning and data augmentation strategies. Our experiments demonstrate that data augmentation can significantly enhance model performance, while transfer learning can effectively prevent overfitting and strengthen generalizability. Compared to existing methods, ACVPred exhibits superior performance and robust generalization both in training and independent test datasets. Moreover, the interpretability study of the model reveals that its transformer-based core can effectively capture key motifs on ACVP sequences, demonstrating strong feature learning capabilities. Additionally, the findings suggest that the sequence feature weights and key motif positions tend to be distributed towards the N-terminal end of ACVP sequences, providing vital clues for the design of ACVPs. In summary, ACVPred is not only a practical and valuable tool for aiding in the design of ACVPs, but its algorithmic concept also serves as an important reference for research on other small sample prediction problems.

A transfer learning strategy for tensile strength prediction in austenitic stainless steel across temperatures

Austenitic stainless steel plays a pivotal role in the nuclear industry with exceptional mechanical properties and corrosion resistance, wherein high-temperature tensile strength (∼290 °C) is a critical factor in ensuring its performance and reliability. This study focuses on the predictive modeling of tensile strength in cast austenitic stainless steel at room and elevated temperatures by developing a transfer learning strategy. The application of a gradient-boosting decision tree algorithm, enhanced by key features derived from room-temperature analyses and augmented with partial high-temperature data, results in significantly increased accuracy. The effective selection of features underscores the stability of critical variables across different temperatures, affirming the efficacy of the proposed transfer learning strategy. The work sheds light on material selection and design for high-temperature applications, and lays the groundwork for future exploration into predictive modeling of material properties in extreme conditions.

Deep transfer learning strategy in intelligent fault diagnosis of rotating machinery

Rotating machinery plays an essential part in many engineering fields. It needs prompt solutions to the prognosis and health management to ensure the system reliability. Facilitated by big data and artificial intelligence, intelligent fault diagnosis provides a new approach. As for the insufficient faulty data and complex conditions, deep transfer learning (DTL) presents a possible approach for cross-domain and cross-machine diagnosis. The published reviews thus far mainly emphasize on the analysis of fault diagnosis based on common classes of DTL or industrial application scenarios. This review concentrates on the applications of DTL in rotating machinery. Moreover, present relevant reviews were mainly till the end of 2021. The latest researches are analyzed from then until now. A special main line based on input types is chosen that distinguishes it from other reviews. From this perspective, it is therefore valuable to comprehensively investigate the fault diagnosis of rotating machinery. This survey firstly outlines the fundamental principle and conventional categories of DTL. The primary applications of DTL in fault diagnosis of rotating machinery are then summarized, and more than 100 relative studies have been analyzed. The special perspective of input types is selected and evaluated, including one-dimensional and two-dimensional, on the DTL framework as applied to the rotary machines discussed. Finally, the existing challenges are pointed out and potential future trends of DTL are prospected. This review helps readers to understand the research status and development trends of transfer intelligent fault diagnosis. It serves to the innovative exploration from multiple different scales.

A Convolution Neural Network-based Approach for Metal Surface Roughness Evaluation

Background: Metal surface roughness detection is an essential step of quality control in the metal processing industry. Due to the high manual involvement and poor efficiency of traditional roughness testing, rapid automated vision detection has received increasing attention in product quality control. Many methods have focused on extracting features related to roughness from images by means of mathematical statistics. However, these methods often rely on extensive experiments and complex calculations, while being sensitive to external environmental disturbances. Methods: In this paper, a convolution neural network-based approach for metal surface roughness evaluation has been proposed. The convolutional neural network was initialized using a transfer learning strategy, and the data augmentation technique was applied to the benchmark dataset for sample expansion. Results: To evaluate this approach, samples of 4 types of roughness classes were prepared. The samples were divided into a training set, validation set, and test set in the ratio of 7:2:1. The accuracy of the neural network on the test set was found to be above 86%. Conclusion: The effectiveness of the proposed approach and its superiority over manual detection have been demonstrated in the experiments.

HybridTransferNet: soil image classification through comprehensive evaluation for crop suggestion

<p>Soil image classification is a critical task within the realms of agriculture and environmental applications. In recent years, the integration of deep learning has sparked significant interest in image-based soil classification. Transfer learning, a well-established technique in image classification, involves finetuning a pre-trained model on a specific dataset. However, conventional transfer learning methods typically focus solely on fine-tuning the final layer of the pre-trained model, which may not suffice to attain high performance on a new task. HybridTransferNet, a unique hybrid transfer learning approach designed for soil classification based on images is proposed in this paper. HybridTransferNet goes beyond the conventional approach by finetuning not only the final layer but also a select number of earlier layers in a pre-trained ResNet50 model. This extension results in substantially enhanced ability to classify when compared to standard transfer learning methods. Our evaluation of HybridTransferNet, conducted on a soil classification dataset, encompasses the reporting of various performance indicators, such as the F1 score, recall, accuracy, and precision. Our findings from experiments highlight HybridTransferNet's advantages over conventional transfer learning strategies, establishing it as a state-of-the-art solution in the domain of soil classification.</p>

Pet Classification Using Transfer Learning

Abstract: This study delves into applying transfer learning to pet classification using pre-trained neural networks, specifically convolutional neural networks (CNNs). It adapts models originally trained on ImageNet to distinguish between different pet species. By fine-tuning these models on a custom pet dataset, the study aims to leverage previous knowledge to improve classification accuracy, even with limited labeled data. Various transfer learning strategies, model architectures, and hyper parameters are analyzed to identify the most effective configuration for accurate and efficient pet classification. The research has implications for pet monitoring, identification, and healthcare, contributing to computer vision and deep learning in specialized image recognition.

Energy-informed graph transformer model for solid mechanical analyses

Physics-informed neural network (PINN) exists some challenges, such as independent and uncorrelated drawbacks leading to convergence impediments, limited interpretability and lack of generalization. In this paper, a novel energy-informed graph transformer model is proposed to overcome the drawbacks of PINN. In the proposed model, the graph neural network-based-attention mechanism is proposed to dynamically calculate weight coefficients between objects of graph-structured data, and then to aggregate weighted combinations of the neighbor objects features to update features of the target objects. The loss function is constructed with homoscedastic uncertainty by introducing trainable scalar parameters, which can be optimized to achieve the best performance of the network as it changes dynamically the topology of the loss function involved in the optimization process. Numerical results show that the proposed method can effectively increase the efficiency, robustness and accuracy of the network approximation of forward and inverse problems of solid mechanics. Furthermore, the proposed model demonstrates excellent generalization capabilities when applied to new problem using transfer learning strategy.

Data-driven strategy for bandgap database construction of perovskites and the potential segregation study

Light-induced segregation limits the practical application of mixed halide perovskites in solar cells. Herein, halide segregation is evaluated by a data-driven approach with constructing a bandgap database of 53,361 mixed ABX3 [where A = Cs, formamidinium (FA) or methylammonium (MA); B = Pb or Sn; X = Br, Cl, or I] perovskites. A transfer learning strategy was employed to fine-tune the parameters of a Graph Neural Network model using experimental and density functional theory (DFT)-calculated bandgaps. This approach accelerated the construction of a unique database, distinguishing it from others primarily focused on ABX3 perovskite element substitution. The database is characterized by continuously varying compositions and accurate bandgaps. It was utilized to calculate the free energy of 20,688 mixed iodine-bromine perovskites and generate corresponding phase diagrams for predicting their light-induced segregation behavior. It is found that the bandgap increases with decreasing ionic radii at the A-site and X-site. This composition-dependent bandgap difference drives halide segregation. Moreover, using a higher Cs content at the A-site, rather than MA, reduces this bandgap difference, enhancing photostability. The proposed data-driven strategy can facilitate the targeted design of novel perovskites with mixed compositions and the investigation of halide perovskite segregation.

Electrocardiography Classification with Leaky Integrate-and-Fire Neurons in an Artificial Neural Network-Inspired Spiking Neural Network Framework.

Monitoring heart conditions through electrocardiography (ECG) has been the cornerstone of identifying cardiac irregularities. Cardiologists often rely on a detailed analysis of ECG recordings to pinpoint deviations that are indicative of heart anomalies. This traditional method, while effective, demands significant expertise and is susceptible to inaccuracies due to its manual nature. In the realm of computational analysis, Artificial Neural Networks (ANNs) have gained prominence across various domains, which can be attributed to their superior analytical capabilities. Conversely, Spiking Neural Networks (SNNs), which mimic the neural activity of the brain more closely through impulse-based processing, have not seen widespread adoption. The challenge lies primarily in the complexity of their training methodologies. Despite this, SNNs offer a promising avenue for energy-efficient computational models capable of displaying a high-level performance. This paper introduces an innovative approach employing SNNs augmented with an attention mechanism to enhance feature recognition in ECG signals. By leveraging the inherent efficiency of SNNs, coupled with the precision of attention modules, this model aims to refine the analysis of cardiac signals. The novel aspect of our methodology involves adapting the learned parameters from ANNs to SNNs using leaky integrate-and-fire (LIF) neurons. This transfer learning strategy not only capitalizes on the strengths of both neural network models but also addresses the training challenges associated with SNNs. The proposed method is evaluated through extensive experiments on two publicly available benchmark ECG datasets. The results show that our model achieves an overall accuracy of 93.8% on the MIT-BIH Arrhythmia dataset and 85.8% on the 2017 PhysioNet Challenge dataset. This advancement underscores the potential of SNNs in the field of medical diagnostics, offering a path towards more accurate, efficient, and less resource-intensive analyses of heart diseases.

GAPS: a geometric attention-based network for peptide binding site identification by the transfer learning approach.

Protein-peptide interactions (PPepIs) are vital to understanding cellular functions, which can facilitate the design of novel drugs. As an essential component in forming a PPepI, protein-peptide binding sites are the basis for understanding the mechanisms involved in PPepIs. Therefore, accurately identifying protein-peptide binding sites becomes a critical task. The traditional experimental methods for researching these binding sites are labor-intensive and time-consuming, and some computational tools have been invented to supplement it. However, these computational tools have limitations in generality or accuracy due to the need for ligand information, complex feature construction, or their reliance on modeling based on amino acid residues. To deal with the drawbacks of these computational algorithms, we describe a geometric attention-based network for peptide binding site identification (GAPS) in this work. The proposed model utilizes geometric feature engineering to construct atom representations and incorporates multiple attention mechanisms to update relevant biological features. In addition, the transfer learning strategy is implemented for leveraging the protein-protein binding sites information to enhance the protein-peptide binding sites recognition capability, taking into account the common structure and biological bias between proteins and peptides. Consequently, GAPS demonstrates the state-of-the-art performance and excellent robustness in this task. Moreover, our model exhibits exceptional performance across several extended experiments including predicting the apo protein-peptide, protein-cyclic peptide and the AlphaFold-predicted protein-peptide binding sites. These results confirm that the GAPS model is a powerful, versatile, stable method suitable for diverse binding site predictions.

Investigating explainable transfer learning for battery lifetime prediction under state transitions

Battery lifetime prediction at early cycles is crucial for researchers and manufacturers to examine product quality and promote technology development. Machine learning has been widely utilized to construct data-driven solutions for high-accuracy predictions. However, the internal mechanisms of batteries are sensitive to many factors, such as charging/discharging protocols, manufacturing/storage conditions, and usage patterns. These factors will induce state transitions, thereby decreasing the prediction accuracy of data-driven approaches. Transfer learning is a promising technique that overcomes this difficulty and achieves accurate predictions by jointly utilizing information from various sources. Hence, we develop two transfer learning methods, Bayesian Model Fusion and Weighted Orthogonal Matching Pursuit, to strategically combine prior knowledge with limited information from the target dataset to achieve superior prediction performance. From our results, our transfer learning methods reduce root-mean-squared error by 41% through adapting to the target domain. Furthermore, the transfer learning strategies identify the variations of impactful features across different sets of batteries and therefore disentangle the battery degradation mechanisms and the root cause of state transitions from the perspective of data mining. These findings suggest that the transfer learning strategies proposed in our work are capable of acquiring knowledge across multiple data sources for solving specialized issues.

Cross-domain structural damage identification using transfer learning strategy

In damage identification of civil structures, training deep neural networks (DNNs) often requires a large volume of annotated training data. However, artificial damage to real-world structures is always forbidden, resulting in insufficient data for training. Transfer learning provides a solution that allows structural damage information in a data-rich domain to be transferred and shared as the prior knowledge to a data-scarce domain, thereby indirectly augmenting available training data in the latter domain. To this end, a multi-task transfer learning strategy has been adopted for the purpose of achieving cross-domain damage identification between a plane frame in the source domain and a three-dimensional frame in the target domain. The strategy can share the learning knowledge from the domain with sufficient training data to the target domain with insufficient data. Thereby, the DNN (named DNN#2) in the target domain can be trained on a small amount of training data. Owing to their ability in data feature extraction, stacked auto-encoders (SAEs) are used to construct the desirable DNN#1 and DNN#2 corresponding to different damage identification tasks, named Task#1 and Task#2, in the two domains. The two DNNs share the hidden layers in order to share damage feature information between the source and target domains. The training datasets of the two domains are first used to jointly pre-train the auto-encoders’ parameters in an unsupervised learning way. Afterwards, a supervised fine-tuning step is carried out to retraining the entire SAEs for better performance. By these means, Task#2 receives some prior knowledge from Task#1, and thus, it is accomplished on limited training data when Task#1 is synchronously achieved. The analysis results demonstrate that the proposed multi-task learning strategy requires only a single training session to simultaneously realize the cross-domain damage identification of two different steel frames.

A Transfer Learning Strategy for Cross-Subject and Cross-Time Hand Gesture Recognition Based on A-Mode Ultrasound

A Transfer Learning Strategy for Cross-Subject and Cross-Time Hand Gesture Recognition Based on A-Mode Ultrasound

Efficient Classification on Remote Sensing Image Using Transfer Learning

With a rapid development in aerial technology, applications of Remote Sensing Images (RSI) have become more diverse. Remote sensing image classification plays a crucial role in analyzing and interpreting Earth observation data for various applications, such as land cover mapping, environmental monitoring, and urban planning. However, accurately classifying remote sensing images poses significant challenges due to their complex spatial and spectral characteristics. In recent years, transfer learning has emerged as a promising technique to improve the classification accuracy by leveraging the knowledge learned from pre-trained models on large-scale datasets. The proposed model explores different transfer learning strategies employed in remote sensing image classification, including fine-tuning, feature extraction, and domain adaptation. It discusses popular pre-trained models, such as VGG16, VGG19, and Inceptionv3, and their applicability to remote sensing datasets. The advantages and limitations of each strategy are analyzed, providing insights into their suitability for various remote sensing applications. A comparative study is done on all these techniques to evaluate the performance measures like Accuracy and Loss.

Web-based diagnostic platform for microorganism-induced deterioration on paper-based cultural relics with iterative training from human feedback

PurposePaper-based artifacts hold significant cultural and social values. However, paper is intrinsically fragile to microorganisms, such as mold, due to its cellulose composition, which can serve as a microorganisms’ nutrient source. Mold not only can damage papers’ structural integrity and pose significant challenges to conservation works but also may subject individuals attending the contaminated artifacts to health risks. Current approaches for strain identification usually require extensive training, prolonged time for analysis, expensive operation costs, and higher risks of secondary damage due to sampling. Thus, in current conservation practices with mold-contaminated artifacts, little pre-screening or strain identification was performed before mold removal, and the cleaning techniques are usually broad-spectrum rather than strain-specific. With deep learning showing promising applications across various domains, this study investigated the feasibility of using a convolutional neural network (CNN) for fast in-situ recognition and classification of mold on paper.MethodsMolds were first non-invasively sampled from ancient Xuan Paper-based Chinese books from the Qing and Ming dynasties. Strains were identified using molecular biology methods and the four most prevalent strains were inoculated on Xuan paper to create mockups for image collection. Microscopic images of the molds as well as their stains situated on paper were collected using a compound microscope and commercial microscope lens for cell phone cameras, which were then used for training CNN models with a transfer learning scheme to perform the classification of mold. To enable involvement and contribution from the research community, a web interface that actuates the process while providing interactive features for users to learn about the information of the classified strain was constructed. Moreover, a feedback functionality in the web interface was embedded for catching potential classification errors, adding additional training images, or introducing new strains, all to refine the generalizability and robustness of the model.Results & ConclusionIn the study, we have constructed a suite of high-confidence classification CNN models for the diagnostic process for mold contamination in conservation. At the same time, a web interface was constructed that allows recurrently refining the model with human feedback through engaging the research community. Overall, the proposed framework opens new avenues for effective and timely identification of mold, thus enabling proactive and targeted mold remediation strategies in conservation.

TL‐GNN: Android Malware Detection Using Transfer Learning

ABSTRACTMalware growth has accelerated due to the widespread use of Android applications. Android smartphone attacks have increased due to the widespread use of these devices. While deep learning models offer high efficiency and accuracy, training them on large and complex datasets is computationally expensive. Hence, a method that effectively detects new malware variants at a low computational cost is required. A transfer learning method to detect Android malware is proposed in this research. Because of transferring known features from a source model that has been trained to a target model, the transfer learning approach reduces the need for new training data and minimizes the need for huge amounts of computational power. We performed many experiments on 1.2 million Android application samples for performance evaluation. In addition, we evaluated how well our framework performed in comparison with traditional deep learning and standard machine learning models. In comparison with state‐of‐the‐art Android malware detection methods, the proposed framework offers improved classification accuracy of 98.87%, a precision of 99.55%, recall of 97.30%, F1‐measure of 99.42%, and a quicker detection rate of 5.14 ms using the transfer learning strategy.

A novel method for quantitative determination of multiple substances using Raman spectroscopy combined with CWT

The identification of mixed solutions is a challenging and important subject in chemical analysis. In this paper, we propose a novel workflow that enables rapid qualitative and quantitative detection of mixed solutions. We use a methanol-ethanol mixed solution as an example to demonstrate the superiority of this workflow. The workflow includes the following steps: (1) converting Raman spectra into Raman images through CWT; (2) using MobileNetV3 as the backbone network, improved multi-label and multi-channel synchronization enables simultaneous prediction of multiple mixture concentrations; and (3) using transfer learning and multi-stage training strategies for training to achieve accurate quantitative analysis. We compare six traditional machine learning algorithms and two deep learning models to evaluate the performance of our new method. The experimental results show that our model has achieved good prediction results when predicting the concentration of methanol and ethanol, and the coefficient of determination R2 is greater than 0.999. At different concentrations, both MAPE and RSD outperform other models, which demonstrates that our workflow has outstanding analytical capabilities. Importantly, we have solved the problem that current quantitative analysis algorithms for Raman spectroscopy are almost unable to accurately predict the concentration of multiple substances simultaneously. In conclusion, it is foreseeable that this non-destructive, automated, and highly accurate workflow can further advance Raman spectroscopy.

Deep learning-driven automated quality assessment of ultra-widefield optical coherence tomography angiography images for diabetic retinopathy

AbstractImage quality assessment (IQA) of fundus images constitutes a foundational step in automated disease analysis. This process is pivotal in supporting the automation of screening, diagnosis, follow-up, and related academic research for diabetic retinopathy (DR). This study introduced a deep learning-based approach for IQA of ultra-widefield optical coherence tomography angiography (UW-OCTA) images of patients with DR. Given the novelty of ultra-widefield technology, its limited prevalence, the high costs associated with equipment and operational training, and concerns regarding ethics and patient privacy, UW-OCTA datasets are notably scarce. To address this, we initially pre-train a vision transformer (ViT) model on a dataset comprising 6 mm × 6 mm OCTA images, enabling the model to acquire a fundamental understanding of OCTA image characteristics and quality indicators. Subsequent fine-tuning on 12 mm × 12 mm UW-OCTA images aims to enhance accuracy in quality assessment. This transfer learning strategy leverages the generic features learned during pre-training and adjusts the model to evaluate UW-OCTA image quality effectively. Experimental results demonstrate that our proposed method achieves superior performance compared to ResNet18, ResNet34, and ResNet50, with an AUC of 0.9026 and a Kappa value of 0.7310. Additionally, ablation studies, including the omission of pre-training on 6 mm × 6 mm OCTA images and the substitution of the backbone network with the ViT base version, resulted in varying degrees of decline in AUC and Kappa values, confirming the efficacy of each module within our methodology.

GraphEGFR: Multi-task and transfer learning based on molecular graph attention mechanism and fingerprints improving inhibitor bioactivity prediction for EGFR family proteins on data scarcity.

The proteins within the human epidermal growth factor receptor (EGFR) family, members of the tyrosine kinase receptor family, play a pivotal role in the molecular mechanisms driving the development of various tumors. Tyrosine kinase inhibitors, key compounds in targeted therapy, encounter challenges in cancer treatment due to emerging drug resistance mutations. Consequently, machine learning has undergone significant evolution to address the challenges of cancer drug discovery related to EGFR family proteins. However, the application of deep learning in this area is hindered by inherent difficulties associated with small-scale data, particularly the risk of overfitting. Moreover, the design of a model architecture that facilitates learning through multi-task and transfer learning, coupled with appropriate molecular representation, poses substantial challenges. In this study, we introduce GraphEGFR, a deep learning regression model designed to enhance molecular representation and model architecture for predicting the bioactivity of inhibitors against both wild-type and mutant EGFR family proteins. GraphEGFR integrates a graph attention mechanism for molecular graphs with deep and convolutional neural networks for molecular fingerprints. We observed that GraphEGFR models employing multi-task and transfer learning strategies generally achieve predictive performance comparable to existing competitive methods. The integration of molecular graphs and fingerprints adeptly captures relationships between atoms and enables both global and local pattern recognition. We further validated potential multi-targeted inhibitors for wild-type and mutant HER1 kinases, exploring key amino acid residues through molecular dynamics simulations to understand molecular interactions. This predictive model offers a robust strategy that could significantly contribute to overcoming the challenges of developing deep learning models for drug discovery with limited data and exploring new frontiers in multi-targeted kinase drug discovery for EGFR family proteins.