Transfer Learning Tasks Research Articles

Fuzzy-membership-based labeling: a new labeling method for both classification task and regression task

In the machine learning and deep learning field, there are two main kinds of tasks: classification and regression. The label for the former is discrete, while for the latter is continuous. Due to the big gaps in labels, these two tasks are generally re solved separately, bringing low training efficiency and waste of computing resources. To this end, this paper proposes a new labeling method based on fuzzy membership. The main idea is to build an intermediate variable, which behaves between continuous and discrete variables. Then, the relation between the intermediate variable and the discrete label can be identified with fuzzy membership. Finally, the fuzzy membership is adopted for building labels to train the source model. After training, the source model can be transferred to achieve both classification and regression tasks. To validate the new labeling method, two typical tasks in the PHM field, aging stage classification and RUL prediction, are selected as the representative for classification and regression tasks, respectively. Furthermore, LSTM with two dense layers is chosen as the benchmark source model. With the C-MAPSS dataset, the superiority of the proposed fuzzy-membership based labeling to improve the network’s task transfer learning performance has been verified.

Fuzzy-Membership-Based Framework for Task Transfer Learning Between Fault Diagnosis and RUL Prediction

Fault classification and remaining useful life (RUL) prediction are two important but different issues in data-driven prognostics and health management. Generally, they are resolved separately in present deep-learning-based methods, with one network orientated for one individual task, bringing low training efficiency and waste of computing resources. Although the output labels for these two tasks are entirely different, one is discrete and the other is continuous, they both reflect the equipment's health status and, thus, are related. To address these two tasks altogether in one deep network, this article proposes a new framework for task transfer learning. Long short-term memory plus two dense layers is built as the source model, and fuzzy membership is first adopted to construct new labels for source model training, with the target to bridge the gap between classical discrete and continuous output labels. Furthermore, three kinds of fuzzy membership functions (triangular, trapezoidal, and Gaussian) are designed for labeling and comparison. To verify the proposed framework, experimental validation is conducted on the C-MAPSS turbo-engine dataset. The degradation phase classification and RUL prediction are designed as discrete classification and continuous regression tasks, respectively. Results confirm that the fuzzy-membership-based labeling can effectively improve the source model performance in task transfer learning. In addition, the trapezoidal performs better than the triangular and Gaussian among the three membership functions.

Automatic patient functionality assessment from multimodal data using deep learning techniques – Development and feasibility evaluation

Wearable devices and mobile sensors enable the real-time collection of an abundant source of physiological and behavioural data unobtrusively. Unlike traditional in-person evaluation or ecological momentary assessment (EMA) questionnaire-based approaches, these data sources open many possibilities in remote patient monitoring. However, defining robust models is challenging due to the data's noisy and frequently missing observations.This work proposes an attention-based Long Short-Term Memory (LSTM) neural network-based pipeline for predicting mobility impairment based on WHODAS 2.0 evaluation from such digital biomarkers. Furthermore, we addressed the missing observation problem by utilising hidden Markov models and the possibility of including information from unlabelled samples via transfer learning. We validated our approach using two wearable/mobile sensor data sets collected in the wild and socio-demographic information about the patients.Our results showed that in the WHODAS 2.0 mobility impairment prediction task, the proposed pipeline outperformed a prior baseline while additionally providing interpretability with attention heatmaps. Moreover, using a much smaller cohort via task transfer learning, the same model could learn to predict generalised anxiety severity accurately based on GAD-7 scores.

AdaTriplet-RA: Domain matching via adaptive triplet and reinforced attention for unsupervised domain adaptation

Unsupervised domain adaptation (UDA) is a transfer learning task where the annotations of the source domain are available, but only have access to the unlabelled target data during training.Previous methods minimise the domain gap by performing distribution alignment between the source and target domains, which has a notable limitation, i.e., at the domain level, but neglecting the sample-level differences, thus preventing the model from achieving superior performance. To solve this, we improve the UDA task with an inter-domain sample-level matching scheme. We apply the widely-used Triplet loss to match the inter-domain samples. To reduce the catastrophic effect of the inaccurate pseudo-labels generated during training, we propose a novel uncertainty measurement method and use this uncertainty to select reliable pseudo-labels automatically. As the selection is uncertainty-aware, the pseudo labels are progressively refined as the training is performed. We apply the advanced Gumbel Softmax technique to realise an adaptive Top-k scheme to achieve adaptive selection. To enable the global ranking optimisation within one batch for the domain matching, the whole model is optimised via a reinforced attention mechanism, using the Average Precision (AP) of the domain matching as the reward.Our model AdaTriplet-RA achieves State-of-the-art results on several public benchmark datasets, and its effectiveness is validated via comprehensive ablation study. Our method improves the accuracy of the baseline by 9.7% (using ResNet-101 as the backbone network) and 6.2% (ResNet-50) on the VisDa dataset and 4.22% (ResNet-50) on the DomainNet dataset. The source code is publicly available at: https://github.com/shuxy0120/AdaTriplet-RA.

Pseudo-label self-training model for transfer learning algorithm

When aligning joint distributions between domains, the existing transfer learning algorithms usually assign pseudo labels due to the lack of labels in target domain. However, the noise in pseudo labels will affect the performance of transfer learning. Pseudo-label self-training for transfer learning (PST-TL) model is proposed to generate reliable pseudo labels for target domain and have a wide range of applications in existing algorithms. Pseudo labels are predicted by an ensemble classifier using absolute majority vote, and labels predicted successfully are considered to be high confidence. The training of ensemble classifier applies the self-training of joint pseudo labels strategy, adding strongly stable data to training set of the classifier. The semi-supervised and unsupervised transfer learning tasks in experiment show that the existing transfer learning algorithm can significantly improve the transfer performance after embedded by PST-TL model.

Interpretable Catalysis Models Using Machine Learning with Spectroscopic Descriptors

The complexity and dynamics of catalytic systems make it challenging to study the catalysts and catalytic reactions. Fortunately, the advance of machine learning (ML) has made descriptor-based catalyst screening and rational design a mainstream research approach. Herein, the spectroscopic descriptors reported in recent years are highlighted in the field of catalysis. Both vibrational spectra and X-ray absorption spectra have demonstrated strong ability to predict catalytic structures and properties. Through several cases, the interpretable ML models based on spectroscopic descriptors are discussed to reveal physical knowledge and catalytic mechanism and to exhibit superiority in transfer learning tasks and imperfect data scenarios. Finally, in this Viewpoint, we illustrate the challenges in the research field of interpretable ML models with spectroscopic descriptors and provide perspectives.

BuresNet: Conditional Bures Metric for Transferable Representation Learning.

As a fundamental manner for learning and cognition, transfer learning has attracted widespread attention in recent years. Typical transfer learning tasks include unsupervised domain adaptation (UDA) and few-shot learning (FSL), which both attempt to sufficiently transfer discriminative knowledge from the training environment to the test environment to improve the model's generalization performance. Previous transfer learning methods usually ignore the potential conditional distribution shift between environments. This leads to the discriminability degradation in the test environments. Therefore, how to construct a learnable and interpretable metric to measure and then reduce the gap between conditional distributions is very important in the literature. In this article, we design the Conditional Kernel Bures (CKB) metric for characterizing conditional distribution discrepancy, and derive an empirical estimation with convergence guarantee. CKB provides a statistical and interpretable approach, under the optimal transportation framework, to understand the knowledge transfer mechanism. It is essentially an extension of optimal transportation from the marginal distributions to the conditional distributions. CKB can be used as a plug-and-play module and placed onto the loss layer in deep networks, thus, it plays the bottleneck role in representation learning. From this perspective, the new method with network architecture is abbreviated as BuresNet, and it can be used extract conditional invariant features for both UDA and FSL tasks. BuresNet can be trained in an end-to-end manner. Extensive experiment results on several benchmark datasets validate the effectiveness of BuresNet.

Joint Transfer Extreme Learning Machine with Cross-Domain Mean Approximation and Output Weight Alignment

With fast learning speed and high accuracy, extreme learning machine (ELM) has achieved great success in pattern recognition and machine learning. Unfortunately, it will fail in the circumstance where plenty of labeled samples for training model are insufficient. The labeled samples are difficult to obtain due to their high cost. In this paper, we solve this problem with transfer learning and propose joint transfer extreme learning machine (JTELM). First, it applies cross-domain mean approximation (CDMA) to minimize the discrepancy between domains, thus obtaining one ELM model. Second, subspace alignment (sa) and weight approximation are together introduced into the output layer to enhance the capability of knowledge transfer and learn another ELM model. Third, the prediction of test samples is dominated by the two learned ELM models. Finally, a series of experiments are carried out to investigate the performance of JTELM, and the results show that it achieves efficiently the task of transfer learning and performs better than the traditional ELM and other transfer or nontransfer learning methods.

Multi-View Feature Transformation Based SVM+ for Computer-Aided Diagnosis of Liver Cancers With Ultrasound Images.

It is feasible to improve the performance of B-mode ultrasound (BUS) based computer-aided diagnosis (CAD) for liver cancers by transferring knowledge from contrast-enhanced ultrasound (CEUS) images. In this work, we propose a novel feature transformation based support vector machine plus (SVM+) algorithm for this transfer learning task by introducing feature transformation into the SVM+ framework (named FSVM+). Specifically, the transformation matrix in FSVM+ is learned to minimize the radius of the enclosing ball of all samples, while the SVM+ is used to maximize the margin between two classes. Moreover, to capture more transferable information from multiple CEUS phase images, a multi-view FSVM+ (MFSVM+) is further developed, which transfers knowledge from three CEUS images from three phases, i.e., arterial phase, portal venous phase, and delayed phase, to the BUS-based CAD model. MFSVM+ innovatively assigns appropriate weights for each CEUS image by calculating the maximum mean discrepancy between a pair of BUS and CEUS images, which can capture the relationship between source and target domains. The experimental results on a bi-modal ultrasound liver cancer dataset demonstrate that MFSVM+ achieves the best classification accuracy of 88.24±1.28%, sensitivity of 88.32±2.88%, specificity of 88.17±2.91%, suggesting its effectiveness in promoting the diagnostic accuracy of BUS-based CAD.

SelfCCL: Curriculum Contrastive Learning by Transferring Self-Taught Knowledge for Fine-Tuning BERT

BERT, the most popular deep learning language model, has yielded breakthrough results in various NLP tasks. However, the semantic representation space learned by BERT has the property of anisotropy. Therefore, BERT needs to be fine-tuned for certain downstream tasks such as Semantic Textual Similarity (STS). To overcome this problem and improve the sentence representation space, some contrastive learning methods have been proposed for fine-tuning BERT. However, existing contrastive learning models do not consider the importance of input triplets in terms of easy and hard negatives during training. In this paper, we propose the SelfCCL: Curriculum Contrastive Learning model by Transferring Self-taught Knowledge for Fine-Tuning BERT, which mimics the two ways that humans learn about the world around them, namely contrastive learning and curriculum learning. The former learns by contrasting similar and dissimilar samples. The latter is inspired by the way humans learn from the simplest concepts to the most complex concepts. Our model also performs this training by transferring self-taught knowledge. That is, the model figures out which triplets are easy or difficult based on previously learned knowledge, and then learns based on those triplets in the order of curriculum using a contrastive objective. We apply our proposed model to the BERT and Sentence BERT(SBERT) frameworks. The evaluation results of SelfCCL on the standard STS and SentEval transfer learning tasks show that using curriculum learning together with contrastive learning increases average performance to some extent.

Combining Human Parsing with Analytical Feature Extraction and Ranking Schemes for High-Generalization Person Reidentification

Person reidentification (re-ID) has been receiving increasing attention in recent years due to its importance for both science and society. Machine learning (particularly Deep Learning (DL)) has become the main re-ID tool that has allowed to achieve unprecedented accuracy levels on benchmark datasets. However, there is a known problem of poor generalization in respect of DL models. That is, models that are trained to achieve high accuracy on one dataset perform poorly on other ones and require re-training. In order to address this issue, we present a model without trainable parameters. This, in turn, results in a great potential for high generalization. This approach combines a fully analytical feature extraction and similarity ranking scheme with DL-based human parsing wherein human parsing is used to obtain the initial subregion classification. We show that such combination, to a high extent, eliminates the drawbacks of existing analytical methods. In addition, we use interpretable color and texture features that have human-readable similarity measures associated with them. In order to verify the proposed method we conduct experiments on Market1501 and CUHK03 datasets, thus achieving a competitive rank-1 accuracy comparable with that of DL models. Most importantly, we show that our method achieves 63.9% and 93.5% rank-1 cross-domain accuracy when applied to transfer learning tasks, while also being completely re-ID dataset agnostic. We also achieve a cross-domain mean average precision (mAP) that is higher than that of DL models in some experiments. Finally, we discuss the potential ways of adding new features to further improve the model. We also show the advantages of interpretable features for the purposes of constructing human-generated queries from verbal descriptions in order to conduct searches without a query image.

A bidirectional semantic segmentation method for remote sensing image based on super-resolution and domain adaptation

ABSTRACT Image semantic segmentation methods based on convolutional neural networks rely on supervised learning with labels, and their performance often drops significantly when applied to unlabelled datasets from different sources. The domain adaptation methods can reduce the inconsistency of feature distribution between the unlabelled target domain data used for testing and the labelled source domain data used for training, thus improve the segmentation performance and have more practical applications. However, in the field of remote sensing image processing, if the spatial resolutions of the source domain and the target domain are different and this problem is not to be solved, the performance of the transferred model will be affected. In this paper, we propose a bidirectional semantic segmentation method based on super-resolution and domain adaption (BSSM-SRDA), which is suitable for the transfer learning task of a semantic segmentation model from a low-resolution source domain data to a high-resolution target domain data. BSSM-SRDA mainly consists of three parts: a shared feature extraction network; a super-resolution image translation module, which incorporates a super-resolution approach to reduce spatial resolution differences and visual style differences of the two domains; a domain-adaptive semantic segmentation module, which combines an adversarial domain adaptation approach to reduce differences at the output level. At the same time, we design a new bidirectional self-supervised learning algorithm for BSSM-SRDA that facilitates mutually beneficial learning of the super-resolution image translation module and the domain-adaptive semantic segmentation module. The experiments demonstrate the superiority of the proposed method over other state-of-the-art methods on two remote-sensing image datasets, with mIoU improvements of 2.5% and 3.2%, respectively. Code: https://github.com/mageliang/BSSM-SRDA.git

A bidirectional trajectory contrastive learning model for driving intention prediction

AbstractDriving intention prediction with trajectory data of surrounding vehicles is critical to advanced driver assistance system for improving the accuracy of decision-making. Previous works mostly focused on trajectory representation based on supervised manners. However, learning generalized and high-quality representations from unlabeled data remains a very challenging task. In this paper, we propose a self-supervised bidirectional trajectory contrastive learning (BTCL) model that learns generalized trajectory representation to improve the performance of the driving intention prediction task. Different trajectory data augmentation strategies and a cross-view trajectory prediction task are constructed jointly as pretext task of contrastive learning. The pretext task can maximize the similarity among different augmentations of the same sample while minimizing similarity among augmentations of different samples. It can not only learn the high-quality representation of trajectory without labeled information but also improve the adversarial attacks on BTCL. Moreover, considering the vehicle trajectory forward and backward follows the same social norms and driving behavior constraints. A bidirectional trajectory contrastive learning module is built to gain more positive samples that further increasing the prediction accuracy in downstream tasks and transfer ability of the model. Experimental results demonstrate that BTCL is competitive with the state-of-the-art, especially for adversarial attack and transfer learning tasks, on real-world HighD and NGSIM datasets.

An Approach to Hyperparameter Tuning in Transfer Learning for Driver Drowsiness Detection Based on Bayesian Optimization and Random Search

Driver drowsiness is a critical factor in road safety, and developing accurate models for detecting it is essential. Transfer learning has been shown to be an effective technique for driver drowsiness detection, as it enables models to leverage large, pre-existing datasets. However, the optimization of hyper-parameters in transfer learning models can be challenging, as it involves a large search space. The core purpose of this research is on presenting an approach to hyperparameter tuning in transfer learning for driving fatigue detection based on Bayesian optimization and Random search algorithms. We examine the efficiency of our approach on a publicly available dataset using transfer learning models with the MobileNetV2, Xception, and VGG19 architectures. We explore the impact of hyperparameters such as dropout rate, activation function, the number of units (the number of dense nodes), optimizer, and learning rate on the transfer learning models’ overall performance. Our experiments show that our approach improves the performance of the transfer learning models, obtaining cutting-edge results on the dataset for all three architectures. We also compare the efficiency of Bayesian optimization and Random search algorithms in terms of their ability to find optimal hyperparameters and indicate that Bayesian optimization is more efficient in finding optimal hyperparameters than Random search. The results of our study provide insights into the importance of hyperparameter tuning for transfer learning-based driver drowsiness detection using different transfer learning models and can guide the selection of hyperparameters and models for future studies in this field. Our proposed approach can be applied to other transfer learning tasks, making it a valuable contribution to the field of ML.

Easy-to-Hard Structure for Remote Sensing Scene Classification in Multitarget Domain Adaptation

Multitarget domain adaptation (MTDA) is a transfer learning task that uses knowledge extracted from a labeled source domain to adapt across multiple unlabeled target domains. The MTDA setting is more complicated than the single-source-single-target domain adaptation (S3TDA) setting because domain shift not only exists in each pair of a source–target domain but also exists among different target domains. In addition, multiple-target domains have their own unique characteristics because they are often collected from various conditions. The semantic information in each target domain can be damaged when they are naïvely merged into a single-target domain. Therefore, the trained model struggles to distinguish between representations in the combined target domain, which degrades the classification performance. Furthermore, the knowledge transferability from the source domain to multiple-target domains in prior studies leaves room for improvement because they only focus on exploiting the relationship of source–target pairs while failing to consider the correlation among multiple-target domains. This article introduces an easy-to-hard adaption structure to solve these problems in MTDA. The proposed method consists of three components: Extracting source representations, Hierarchical intratarget feature Alignment, and Collaborative intertarget feature Alignment, called EHACA. These components are used to encode the semantic information in each target domain and explore the relationships between the source and target domains, and among different target domains. The proposed method shows outstanding classification performance over five remote sensing datasets of MTDA tasks, surpassing state-of-the-art approaches in most experimental scenarios.

A systematic study of transfer learning for colorectal cancer detection

Background and objectiveWith the rapid development of data science methods like deep learning, these methods have already been used into the field of healthcare and medicine. However, due to regulations and ethical issues, it is practically difficult to obtain large amount of medical data for training deep learning models. Transfer learning is a powerful tool to reuse the knowledge gained from different domains, which makes it possible to retrain deep learning models only with small datasets in medical image processing. In this contribution, a systematic study of model transfer techniques like fine tuning parts of the network or adding additional layers, for medical image data was conducted. MethodsThe study accomplished a binary classification task based on a colorectal cancer dataset, including microsatellite unstable or hypermutated (MSIMUT) and microsatellite stable (MSS) images. By using K-fold cross-validation, the performances of five pretrained models (DenseNet121, DenseNet201, InceptionV3, MobileNetV2 and ResNet50) were assessed according to balanced accuracy. As baseline methods, combinations of transfer learning as feature extractor and principal component analysis with linear discriminant analysis (PCA-LDA) or support vector machine (PCA-SVM) were utilised, and compared with the transfer learning counterparts. ConclusionsThe results have shown that adding convolutional layers perform obviously better than simply using the original network or fine-tuning some last layers of the network. Furthermore, a proposed bagging method performed well on a testing dataset. This study reduces the workload for future transfer learning tasks in the biomedical domain and allows to test promising transfer learning strategies first.

Task Transfer Learning for Prediction of Transient Nitrogen Oxides, Soot, and Total Hydrocarbon Emissions of a Diesel Engine

Task Transfer Learning for Prediction of Transient Nitrogen Oxides, Soot, and Total Hydrocarbon Emissions of a Diesel Engine

Identification of asynchronous motor and transformer situations in thermal images by utilizing transfer learning-based deep learning architectures

Asynchronous motors are thoroughly preferred in industrial applications regarding their advantages in comparison with other motor types, and transformers constitute another oft-used category for the adjustment of voltage to be fed to an electrical system. Concerning the inevitable usage of these equipment, the fault diagnostics are generally fulfilled by in-depth determinations of stator current signals, magnetic flux distributions, etc. which require conventional electrical measurements. Herein, thermal image analyses arise as an easy way to identify the situations of electrical equipment in which there is no need for direct intervention to the structure. In this paper, we handle the thermal image-based analyses to distinguish the situations of asynchronous motors and transformers. For this purpose, without a pre-processing step, efficient deep learning architectures (DenseNet201, MobileNetV2, ResNet50, ShuffleNet, Xception) are examined to discriminate twenty situations formed by combining the conditions of both types of equipment. These conditions are defined as the cooling fan failure, rotor fault, short-circuit faults in different phases and in various rates inside the motor, short-circuit faults in different rates inside the transformer, no-load motor, and no-load transformer. In experiments, the hyperparameters of models are examined in a comprehensive manner to observe the highest scores of architectures that can be achieved. Herein, four phenomena (mini-batch size, learning rate, learning rate drop factor, optimizer type) are evaluated to perform the transfer learning task, not to spoil the main part of the models, and to reveal the appropriate model for thermal image classification. In trials, an 80–20% training-test split is allowed to compare the models without data augmentation. As a result, the highest performance is observed with all deep learning architectures by attaining 100% accuracy for condition discrepancy of thermal images belonging to the three phase-motor and one phase-transformer. In addition to the accuracy-based analyses, an in-depth evaluation is presented to reveal the most appropriate architecture in thermal image classification.

Transfer learning data adaptation using conflation of low‐level textural features

AbstractAdapting the target dataset for a pre‐trained model is still challenging. These adaptation problems result from a lack of adequate transfer of traits from the source dataset; this often leads to poor model performance resulting in trial and error in selecting the best‐performing pre‐trained model. This paper introduces the conflation of source domain low‐level textural features extracted using the first layer of the pre‐trained model. The extracted features are compared to the conflated low‐level features of the target dataset to select a higher‐quality target dataset for improved pre‐trained model performance and adaptation. From comparing the various probability distance metrics, Kullback‐Leibler is adopted to compare the samples from both domains. We experiment on three publicly available datasets and two ImageNet pre‐trained models used in past studies for results comparisons. This proposed approach method yields two categories of the target samples with those with lower Kullback‐Leibler values giving better accuracy, precision and recall. The samples with the lower Kullback‐Leibler values give a higher margin accuracy rate of 0.22%–9.15%, thereby leading to better model adaptation and easier model selection process for the target transfer learning datasets and tasks.

Dynamic fine‐tuning layer selection using Kullback–Leibler divergence

AbstractThe selection of layers in the transfer learning fine‐tuning process ensures a pre‐trained model's accuracy and adaptation in a new target domain. However, the selection process is still manual and without clearly defined criteria. If the wrong layers in a neural network are selected and used, it could lead to poor accuracy and model generalization in the target domain. This paper introduces the use of Kullback–Leibler divergence on the weight correlations of the model's convolutional neural network layers. The approach identifies the positive and negative weights in the ImageNet initial weights selecting the best‐suited layers of the network depending on the correlation divergence. We experiment on four publicly available datasets and six ImageNet pre‐trained models used in past studies for results comparisons. This proposed approach method yields better accuracies than the standard fine‐tuning baselines with a margin accuracy rate of 10.8%–24%, thereby leading to better model adaptation for target transfer learning tasks.