Neural Network Based Transfer Learning Research Articles

Neural network–based transfer learning to improve stiffness modeling of industrial robots with small experimental data sets

AbstractStiffness modeling is an essential subject for the composition of robot control. Accurate stiffness modeling is helpful for improving the control accuracy of industrial robots, particularly under dynamic load circumstances. The classic virtual joint modeling (VJM) method is challenging in predicting the deformation of the end-effector throughout the full workspace due to the nonlinear deformation of the robot joint and its serial articulated structure. This paper proposes a full-space stiffness modeling method for robots based on the integration of a multi-layer perceptual (MLP) model and VJM. To provide enough training data for the MLP model, VJM is used to build a stiffness model with a small set of experimental data to generate 106,400 training data. A model-based transfer learning approach is proposed to improve the model’s accuracy and generalization regarding the difference between generated training data and actual experimental data. The VJM stiffness model is compared with the MLP stiffness model and the existing CNN-based transfer learning model based on the same experimental data. Considering the deformation prediction in the three directions in Cartesian space, the mean absolute error, standard deviation, and maximum error of the MLP model are decreased by at least 24.90%, 14.20%, and 8.50%, respectively, than the VJM. These prediction results demonstrate that the proposed modeling technique can significantly increase the accuracy of robot stiffness modeling, which is essential for position compensation in precise motion control of robots under dynamic load.

Fractional whale driving training-based optimization enabled transfer learning for detecting autism spectrum disorder

Autism Spectrum Disorder (ASD) is a neurological illness that degrades communication and interaction among others. Autism can be detected at any stage. Early detection of ASD is important in preventing the communication, interaction and behavioral outcomes of individuals. Hence, this research introduced the Fractional Whale-driving Driving Training-based Based Optimization with Convolutional Neural Network-based Transfer learning (FWDTBO-CNN_TL) for identifying ASD. Here, the FWDTBO is modelled by the incorporation of Fractional calculus (FC), Whale optimization algorithm (WOA) and Driving Training-based Optimization (DTBO) that trains the hyperparameters of CNN-TL. Moreover, the Convolutional Neural Networks (CNN) utilize the hyperparameters from trained models, like Alex Net and Shuffle Net in such a way that the CNN-TL is designed. To improve the detection efficiency, the nub region was extracted and carried out with the functional connectivity-based Whale Driving Training Optimization (WDTBO) algorithm. Moreover, the TL is tuned by the FWDTBO algorithm. The result reveals that the ASD detection technique, FWDTBO-CNN-TL acquired 90.7 % accuracy, 95.4 % sensitivity, 93.7 % specificity and 93 % f-measure with the ABIDE-II dataset.

Deep transfer learning strategy in intelligent fault diagnosis of gas turbines based on the Koopman operator

The gas turbine engine is a predominant prime mover in the transport and energy sectors, and ensuring its reliable operation holds paramount significance. While intelligent fault diagnosis (FD) approaches have seen successful advancements within the gas turbine FD landscape, many existing methods operate under the assumption of identical health states during both data collection and the FD process. Moreover, most previous studies have overlooked the diagnosis of both sensors and actuators. Another critical challenge lies in isolating simultaneous and multiple faults and providing compromising FD performance, especially in the face of continued system performance degradation. Aiming at these problems, this study develops a novel unsupervised data-driven FD strategy based on leveraging the potential of transfer learning and the Koopman operator. A deep neural network-based transfer learning framework is proposed for realizing a precise adaptive linear model called the deep transfer linear (DTL) model enabling reliable prediction of the system’s behavior in various situations and designing structured fault residuals. To this end, a deep neural network framework is used to obtain a precise Koopman model realization using the richly collected data in the source domain. Subsequently, the realized model is fine-tuned for the target domains associated with the degraded system, mitigating the adverse effects of domain shift and addressing the rich data scarcity problem in the target domain. In addition, the dedicated and generalized residual sets are designed and generated employing the geometric approach for fault isolation and a decision-making analysis is developed to diagnose simultaneous faults. The reliability of the proposed strategy is demonstrated through various experiments in the presence of noise and performance degradation, and a comparative performance analysis is conducted between the proposed strategy and another data-driven method showcasing the superiority of the proposed approach.

Detecting facial image forgeries with transfer learning techniques

Digital images have become ubiquitous in our daily lives, appearing on our smartphone screens and online websites. They are widely used in numerous industries, including media, forensic and criminal investigations, medicine, and more. The ease of access to consumer photo editing tools has made it simple to manipulate images. However, such altered images pose a serious risk in fields where image authenticity is crucial, making it challenging to confirm the reliability of digital images. Digital image fraud involves altering an image's meaning without leaving any obvious signs. In this study, we present three convolutional neural network-based transfer learning techniques “CNN” classification of facial image forgeries, using VGG-19, InceptionV3, and DenseNet201. Among these methods, DenseNet201 achieved the highest accuracy of 99%, followed by InceptionV3 at 94% and VGG-19 at 84%.

DEEP LEARNING-BASED FEATURE FUSION AND TRANSFER LEARNING FOR APPROXIMATING pIC VALUE OF COVID-19 MEDICINE USING DRUG DISCOVERY DATA

The pandemic disease Coronavirus 2019 (COVID-19) caused thousands of infections and deaths globally. It is important to introduce new medicines to address the critical situation in the medical system. The determination of approximate pIC value is necessary for designing medicines based on molecular compounds. Generally, the approximation of pIC value is a lengthy process, so it is difficult and time-consuming. Hence it is essential to introduce a new technique for automatic approximation. In this research, a Convolutional Neural Network-based transfer learning (CNN-TL) is designed for approximating the pIC value. Initially, Simplified Molecular Input Line Entry System (SMILES) notation is extracted from SMILES string symbols using an entropy-based one-hot encoding matrix and the molecular formula-based encoding. The molecular features are then extracted from the input data using Lorentzian similarity and Deep Residual Network (DRN). The pIC value approximation is performed using the CNN-TL model, where the Visual Geometry Group Network-16 (VGGNet-16) is used to fetch hyperparameters used to initialize the CNN. The experimental results proved that the designed CNN-TL technique achieved minimum error rates with normalized values of 0.406 for R2, 0.516 for Root Mean Square Error (RMSE), 0.267 for Mean Square Error (MSE), and for 0.277 Mean Absolute Percentage Error (MAPE).

Transfer learning based remaining useful life prediction of lithium-ion battery considering capacity regeneration phenomenon

Lithium-ion battery (LIB) has been widely used in various energy storage systems, and the accurate remaining useful life (RUL) prediction for LIB is critical to ensure the normal operation of system. However, the capacity regeneration (CR) phenomenon caused by the non-working state of LIB will seriously affect the capacity degradation trajectory of LIB, thus leading to an inaccurate RUL prediction result. To solve this problem, this paper proposes a non-parametric CR detection algorithm to detect the CR phenomenon online and quantitatively correct the prediction errors caused by different CR phenomena. In addition, a data reconstruction algorithm and a parameter update scheme are proposed to solve the problem of unit-to-unit difference among LIBs. Finally, the long and short term memory neural network-based transfer learning model is applied to predict the RUL of target LIB. The effectiveness of the proposed model is verified by the real LIB dataset of NASA. Compared with some existing RUL prediction models considering CR phenomenon, the proposed model exhibits higher accuracy and better universality.

Novel skin cancer Detection based transfer learning with optimization algorithm using Dermatology Dataset

Detecting skin cancer at the preliminary stage is a challenging issue, and is of high significance for the affected patients. Here, Fractional Gazelle Optimization Algorithm_Convolutional Neural Network based Transfer Learning with Visual Geometric Group-16 (FGOA_CNN based TL with VGG-16) is introduced for primary prediction of skin cancer. Initially, input skin data is acquired from the database and it is fed to the data preprocessing. Here, data preprocessing is done by missing value imputation and linear normalization. Once data is preprocessed, the feature selection is done by the proposed FGOA. Here, the proposed FGOA is an integration of Fractional Calculus (FC) and Gazelle Optimization Algorithm (GOA). After that, skin cancer detection is carried out using CNN-based TL with VGG-16, which is trained by the proposed FGOA and it is an integration of FC and GOA. Moreover, the efficiency of the proposed FGOA_ CNN-based TL with VGG-16 is examined based on five various metrics, like accuracy, Positive Predictive Value (PPV), True Positive Rate (TPR), True Negative Rate (TNR), and Negative Predictive Value (NPV) and the outcome of experimentation reveals that the devised work is highly superior and has attained maximal values of metrics is 92.65%, 90.35%, 91.48%, 93.56%, 90.77% respectively.

A Comparative Analysis of Convolutional Neural Networkbased Transfer Learning Models for Plant Disease Detection

Agriculture holds a significant role in the economy of many countries by providing food security, employment opportunities, and contributing to the overall economic growth of the nation. However, the spread of plant diseases can pose a significant threat to crop production and result in reduced yields, food scarcity, and ecological imbalances. The current manual process of diagnosing plant diseases is not only time-consuming but also prone to errors, relying heavily on the expertise of pathologists. To tackle this challenge, this paper presents a user-friendly and accessible machine learning-based plant disease detection system that uses Convolutional Neural Network (CNN) based transfer-learning models to detect and categorize plants as either healthy or diseased and provides information on the precautions for the identified disease. By comparing the performance of three cutting-edge CNN architectures, VGG16, ResNet50, and EfficientNetV2S, on publicly available plant disease dataset, this paper aims to determine a model for detecting and classifying various plant diseases. We found that ResNet50 achieves the highest accuracy of 96.5%. This proposed system can enable timely and accurate disease detection, helping farmers take preventive measures and reducing the risk of crop loss.

Toward Production Forecasting for Shale Gas Wells Using Transfer Learning

Accurate prediction of shale gas well production and estimated ultimate recovery (EUR) is always a difficult and hot spot in shale gas development. In particular, the production and EUR prediction of shale gas wells in new production blocks are faced with the lack of field gas well data and the difficulty of model development. In view of the above problems, this study proposes a new deep transfer learning strategy, which uses transfer component analysis (TCA) and deep neural network (DNN) to achieve shale gas well production and EUR prediction across formations/blocks. The feature extractor based on TCA can narrow the input feature distribution of the source and the target domains. The neural network model can be used to establish a domain-adaptive transfer learning model without the prediction performance degradation caused by distribution offset. Validity and accuracy of the model were analyzed using gas well data from Weiyuan and Luzhou blocks in Sichuan Basin, China. The results appear that the reasonable application of TCA can greatly improve the prediction performance of shale gas well transfer learning model. For data sets of the same size, compared with the transfer learning model developed by classical machine learning algorithms, the proposed neural network-based transfer learning model can significantly improve the accuracy of production prediction across formations/blocks. In addition, the proposed model can also be extended to other types of oil and gas production prediction tasks cross formations/blocks.

Energy efficiency and savings analysis with multirate sampling for petrochemical process using convolutional neural network-based transfer learning

Energy efficiency in the petrochemical industry is crucial in reducing energy consumption and environmental impact. An accurate energy efficiency model will provide valuable insight for supporting operational adjustment decisions. In practice, due to inconsistent sampling intervals in the petrochemical industry, the traditional approach for obtaining energy efficiency may be unreliable and difficult to handle these multirate data characteristics. Therefore, in this paper, a multi-channel convolutional neural network model integrating a model parameter-based transfer learning approach is proposed to improve the prediction of energy efficiency under inconsistent sampling intervals. The multi-channel structure aims to recognize a different pattern from the dataset by convolving the information along the time dimension. Concurrently, transfer learning allows the model to learn a new pattern of input after the model is fully trained. Finally, the performance for energy efficiency prediction and saving analysis is validated by applying it to the vinyl chloride monomer production case study. The result shows that the proposed model outperformed traditional models and typical convolutional neural network structures in terms of accuracy and reproducibility, with an r-square of 0.97. The utilization of transfer learning prevents a significant drop in performance and enhances adaptability in model learning on real-time energy tracking. Moreover, the energy gap analysis of the prediction result identified a significant energy-saving potential, which would decrease annual energy consumption by 7.25% on average and a 5,709-ton reduction in carbon dioxide emissions.

Internal multiple suppression with convolutional neural network-based transfer learning

Abstract Deep learning methods have been widely applied to seismic denoising, outperforming existing methods in efficiency and generalization. For internal multiple suppression, however, deep learning models face the challenge of low generalization owing to the variability of internal multiples. This diminishes the advantages of deep learning methods over current processing flows. To overcome this, we redesign a convolutional neural network (CNN) method in the aspect of label generation and training process to suppress internal multiples. We apply the virtual event (VE) method to a small amount of data and take removed internal multiple data as labels to accelerate the network training, which is multiple learning. Instead of training a pervasive model to apply to all datasets, we rely on transfer learning to generalize. We finetune the synthetic data training model on target datasets to obtain the model applicable to the dataset with a low requirement of training data and time. Tests on synthetic and field data demonstrate the effects of multiple learning and transfer learning, as well as the competitive demultipling performance of our method compared with both the VE method and the original CNN in efficiency and primary-preserving ability.

MDTL: A Novel and Model-Agnostic Transfer Learning Strategy for Cross-Subject Motor Imagery BCI.

In recent years, deep neural network-based transfer learning (TL) has shown outstanding performance in EEG-based motor imagery (MI) brain-computer interface (BCI). However, due to the long preparation for pre-trained models and the arbitrariness of source domain selection, using deep transfer learning on different datasets and models is still challenging. In this paper, we proposed a multi-direction transfer learning (MDTL) strategy for cross-subject MI EEG-based BCI. This strategy utilizes data from multi-source domains to the target domain as well as from one multi-source domain to another multi-source domain. This strategy is model-independent so that it can be quickly deployed on existing models. Three generic deep learning models for MI classification (DeepConvNet, ShallowConvNet, and EEGNet) and two public motor imagery datasets (BCIC IV dataset 2a and Lee2019) are used in this study to verify the proposed strategy. For the four-classes dataset BCIC IV dataset 2a, the proposed MDTL achieves 80.86%, 81.95%, and 75.00% mean prediction accuracy using the three models, which outperforms those without MDTL by 5.79%, 6.64%, and 11.42%. For the binary-classes dataset Lee2019, MDTL achieves 88.2% mean accuracy using the model DeepConvNet. It outperforms the accuracy without MDTL by 23.48%. The achieved 81.95% and 88.2% are also better than the existing deep transfer learning strategy. Besides, the training time of MDTL is reduced by 93.94%. MDTL is an easy-to-deploy, scalable and reliable transfer learning strategy for existing deep learning models, which significantly improves model performance and reduces preparation time without changing model architecture.

A CNN-based transfer learning method for leakage detection of pipeline under multiple working conditions with AE signals

Pipeline leakage detection is a crucial part of pipeline integrity management. Acoustic emission (AE) based leakage detection is widely used in this field. The latest detection methods are combined AE with convolutional neural networks. However, these methods are often confined to the complex signal processing and computing power and only target specific working conditions. To address these issues, this study proposes a convolutional neural network-based transfer learning (CNN-TL) method for pipeline leakage detection under multiple working conditions. Seven AE datasets are collected from pipeline leakage experiments under different work environments, transporting medium, and fluid pressure. The proposed method converted raw AE signals into three-channel images by a novel conversion method, which avoids reliance on expert knowledge and complex signal processing. CNN-TL is investigated by two different approaches, feature-based CNN-TL and parameter-based CNN-TL. The following nine pre-trained CNN models are used to select the optimal CNN-TL model: Alexnet, Squeezenet, Vgg19, Googlenet, Inceptionv3, Mobilenetv2, Xception, Resnet101, and Densenet201. Results show that the proposed feature-based CNN-TL method significantly outperformed parameter-based CNN-TL and traditional CNN methods, especially on two-phase flow datasets. The highest accuracy of seven AE datasets obtained by the feature-based CNN-TL methods are 100.00%, 100.00%, 100.00%, 99.33%, 85.67%, 87.67%, 74.33%, 83.33% respectively. Moreover, the computation time of proposed method is 16.78 s on average by using the best layers in feature-based CNN-TL. It can be concluded that the proposed method does not rely on signal processing, requires less computational power, and can accomplish accurate detection of pipeline leaks under multiple working conditions.

Parameter-efficient feature-based transfer for paraphrase identification

Abstract There are many types of approaches for Paraphrase Identification (PI), an NLP task of determining whether a sentence pair has equivalent semantics. Traditional approaches mainly consist of unsupervised learning and feature engineering, which are computationally inexpensive. However, their task performance is moderate nowadays. To seek a method that can preserve the low computational costs of traditional approaches but yield better task performance, we take an investigation into neural network-based transfer learning approaches. We discover that by improving the usage of parameters efficiently for feature-based transfer, our research goal can be accomplished. Regarding the improvement, we propose a pre-trained task-specific architecture. The fixed parameters of the pre-trained architecture can be shared by multiple classifiers with small additional parameters. As a result, the computational cost left involving parameter update is only generated from classifier-tuning: the features output from the architecture combined with lexical overlap features are fed into a single classifier for tuning. Furthermore, the pre-trained task-specific architecture can be applied to natural language inference and semantic textual similarity tasks as well. Such technical novelty leads to slight consumption of computational and memory resources for each task and is also conducive to power-efficient continual learning. The experimental results show that our proposed method is competitive with adapter-BERT (a parameter-efficient fine-tuning approach) over some tasks while consuming only 16% trainable parameters and saving 69-96% time for parameter update.

Ultrasound image intelligent diagnosis in community-acquired pneumonia of children using convolutional neural network-based transfer learning.

Studies show that lung ultrasound (LUS) can accurately diagnose community-acquired pneumonia (CAP) and keep children away from radiation, however, it takes a long time and requires experienced doctors. Therefore, a robust, automatic and computer-based diagnosis of LUS is essential. To construct and analyze convolutional neural networks (CNNs) based on transfer learning (TL) to explore the feasibility of ultrasound image diagnosis and grading in CAP of children. 89 children expected to receive a diagnosis of CAP were prospectively enrolled. Clinical data were collected, a LUS images database was established comprising 916 LUS images, and the diagnostic values of LUS in CAP were analyzed. We employed pre-trained models (AlexNet, VGG 16, VGG 19, Inception v3, ResNet 18, ResNet 50, DenseNet 121 and DenseNet 201) to perform CAP diagnosis and grading on the LUS database and evaluated the performance of each model. Among the 89 children, 24 were in the non-CAP group, and 65 were finally diagnosed with CAP, including 44 in the mild group and 21 in the severe group. LUS was highly consistent with clinical diagnosis, CXR and chest CT (kappa values = 0.943, 0.837, 0.835). Experimental results revealed that, after k-fold cross-validation, Inception v3 obtained the best diagnosis accuracy, PPV, sensitivity and AUC of 0.87 ± 0.02, 0.90 ± 0.03, 0.92 ± 0.04 and 0.82 ± 0.04, respectively, for our dataset out of all pre-trained models. As a result, best accuracy, PPV and specificity of 0.75 ± 0.03, 0.89 ± 0.05 and 0.80 ± 0.10 were achieved for severity classification in Inception v3. LUS is a reliable method for diagnosing CAP in children. Experiments showed that, after transfer learning, the CNN models successfully diagnosed and classified LUS of CAP in children; of these, the Inception v3 achieves the best performance and may serve as a tool for the further research and development of AI automatic diagnosis LUS system in clinical applications. www.chictr.org.cn ChiCTR2200057328.

An efficient approach for guided wave structural monitoring of switch rails via deep convolutional neural network-based transfer learning

Data-driven approaches in structural health monitoring have received increasing attention, especially advances in deep learning-based methods, which have further driven the development of data-driven damage detection. Due to the limited availability of guided wave samples and the imbalance between data classes, this study proposes a deep convolutional neural network-based transfer learning (DCTL) approach for the structure monitoring of switch rails using guided wave monitoring signals. A pretrained model based on Inception-ResNet-V2 was adopted and fine-tuned. Different methods for converting 1D signals into 2D images were investigated to find the optimal approach that meets practical monitoring requirements. Affine transformations were used for data augmentation to improve generalization ability and to avoid the overfitting of the training model. Two types of guided wave monitoring experiments on the foot and web of switch rails were conducted to evaluate the proposed method against different conventional methods in the field of switch rails. In addition, the DCTL method was investigated, with and without pretrained weights, along with different frozen layers. The classification results show that the proposed method can identify damage in challenging situations and outperforms conventional methods.

A Color-Texture-Based Deep Neural Network Technique to Detect Face Spoofing Attacks

Abstract Given the face spoofing attack, adequate protection of human identity through face has become a significant challenge globally. Face spoofing is an act of presenting a recaptured frame before the verification device to gain illegal access on behalf of a legitimate person with or without their concern. Several methods have been proposed to detect face spoofing attacks over the last decade. However, these methods only consider the luminance information, reflecting poor discrimination of spoofed face from the genuine face. This article proposes a practical approach combining Local Binary Patterns (LBP) and convolutional neural network-based transfer learning models to extract low-level and high-level features. This paper analyzes three color spaces (i.e., RGB, HSV, and YCrCb) to understand the impact of the color distribution on real and spoofed faces for the NUAA benchmark dataset. In-depth analysis of experimental results and comparison with other existing approaches show the superiority and effectiveness of our proposed models.

Feature space transformation of user-clicks and deep transfer learning framework for fraudulent publisher detection in online advertising

The concept drift is a challenge in click fraud detection wherein frequent changes in the actual status label of publishers complicate the identification of publishers’ fraudulent behavior. However, using transfer learning by leveraging the knowledge from previously learned domains to newer domains can make these differences more accessible while saving training time and improving the model’s performance. But the absence of other user-click datasets available publicly poses complexity in using transfer learning. Therefore, to use transfer learning towards predicting the publisher’s conduct concerning change in their labels, this work aims to transform 1D user-click non-image features into a 2D graphical image. The work proposes a deep convolution neural network-based transfer learning (DCNNTr) framework that utilizes different pre-trained Deep Convolutional Neural Network (DCNN) models as powerful feature extractors that leverage prior learnings to avert learning from scratch. The robust features extracted by the feature extractors help identify the conduct of publishers and classify them as fraudulent or non-fraudulent from 2D graphical images using machine learning models. By leveraging the weighted layers in extracting features, DCNN models utilize their special properties of being computationally efficient and locally focused. We evaluated the designed model on the FDMA2012 user-click dataset using precision, recall, F1-score, and AUC. Current work uniquely transforms the time series user-click non-image data into an image form. The experimental results demonstrate that features extracted with DenseNet121 followed by GTB have identified the fraudulent publishers with an average precision score of 79.8%.

Transfer learning approach toward joint monitoring of bit rate and modulation format.

Convolutional neural network based transfer learning (TL) is proposed to achieve joint optical performance monitoring with bit rate and modulation format identification in optical communication systems. TL is used to improve the execution of various tasks by extracting features without knowing other optical link parameters. Eye diagrams of four different modulation formats are generated at optical signal-to-noise ratios (OSNRs) varying from 15 to 30 dB for two distinct bit rates, which are then identified simultaneously with a trained deep neural network. In addition, comparisons of different TL approaches are presented. The database is divided into distinct categories with varying parameter ranges in offline mode, and prediction models are assigned to each class. The results suggest that the proposed system may greatly increase identification performance over existing strategies by utilizing TL techniques. The impacts of training, testing, and validation data size, as well as model structure based on TL, are also thoroughly investigated. The results reveal that the VGG16 achieves the highest accuracies compared to other deep learning algorithms even at low OSNR values of 20 dB. The proposed structure can intelligently evaluate the signals of future heterogeneous optical communications, and the results can be used to enhance optical network management.

Neural Network Based Transfer Learning for Robot Path Generation

Abstract In this paper, an artificial neural network (ANN)-based transfer learning approach of inverse displacement analysis of robot manipulators is studied. ANNs with different structures are applied utilizing data from different end-effector paths of a manipulator for training purposes. Four transfer learning methods are proposed by applying pretrained initial parameters. Final training results of ANN with transfer learning are compared with those of ANN with random initialization. To investigate the rate of convergence of data fitting comprehensively, different values of performance targets are defined. The computing epochs and performance measures are compared. It is presented that, depending on the structure of ANN, the proposed transfer learning methods can accelerate the training process and achieve higher accuracy. Depending on the method, the transfer learning improves the performance differently.