Semi-supervised Transfer Learning Research Articles

Enhanced vision-transformer integrating with semi-supervised transfer learning for state of health and remaining useful life estimation of lithium-ion batteries

The state of health (SOH) and remaining useful life (RUL) of lithium-ion batteries are crucial for health management and diagnosis. However, most data-driven estimation methods heavily rely on scarce labeled data, while traditional transfer learning faces challenges in handling domain shifts across various battery types. This paper proposes an enhanced vision-transformer integrating with semi-supervised transfer learning for SOH and RUL estimation of lithium-ion batteries. A depth-wise separable convolutional vision-transformer is developed to extract local aging details with depth-wise convolutions and establishes global dependencies between aging information using multi-head attention. Maximum mean discrepancy is employed to initially reduce the distribution difference between the source and target domains, providing a superior starting point for fine-tuning the target domain model. Subsequently, the abundant aging data of the same type as the target battery are labeled through semi-supervised learning, compensating for the source model's limitations in capturing target battery aging characteristics. Consistency regularization incorporates the cross-entropy between predictions with and without adversarial perturbations into the gradient backpropagation of the overall model. In the evaluation across different types of batteries, the mean absolute error of SOH and RUL estimation did not exceed 1.4% and 4%, respectively. Leveraging extensive unlabeled aging data, the proposed method could achieve accurate estimation of SOH and RUL.

SAFENESS: A Semi-Supervised Transfer Learning Approach for Sea State Estimation Using Ship Motion Data

SAFENESS: A Semi-Supervised Transfer Learning Approach for Sea State Estimation Using Ship Motion Data

Semi-Supervised Transfer Learning Framework for Aging-Aware Library Characterization

Semi-Supervised Transfer Learning Framework for Aging-Aware Library Characterization

STFuse: Infrared and Visible Image Fusion via Semisupervised Transfer Learning.

Infrared and visible image fusion (IVIF) aims to obtain an image that contains complementary information about the source images. However, it is challenging to define complementary information between source images in the lack of ground truth and without borrowing prior knowledge. Therefore, we propose a semisupervised transfer learning-based method for IVIF, termed STFuse, which aims to transfer knowledge from an informative source domain to a target domain, thus breaking the above limitations. The critical aspect of our method is to borrow supervised knowledge from the multifocus image fusion (MFIF) task and to filter out task-specific attribute knowledge by using a guidance loss Lg , which motivates its cross-task use in IVIF tasks. Using this cross-task knowledge effectively alleviates the limitation of the lack of ground truth on fusion performance, and the complementary expression ability under the constraint of supervised knowledge is more instructive than prior knowledge. Moreover, we designed a cross-feature enhancement module (CEM) that utilizes self-attention and mutual-attention features to guide each branch to refine features and then facilitate the integration of cross-modal complementary features. Extensive experiments demonstrate that our method has good advantages in terms of visual quality and statistical metrics, as well as the docking of high-level vision tasks, compared with other state-of-the-art methods.

An active semi-supervised transfer learning method for robot pose error prediction and compensation

Robots are widely employed in industrial settings owing to their efficiency, flexibility, and extensive operational ranges. However, their application in high-precision scenarios is limited owing to their low absolute accuracies. Existing methods suffer from high measurement costs, and limited applicability and accuracy. To address these issues, an active semi supervised transfer learning method (ASTL) is introduced. The pose error prediction problem was modelled as a transfer learning paradigm for the first time. It leverages the proposed multi-stage greedy sampling (MGS) for informed sample labelling combined with coarse calibration and semi supervised transfer learning (STL) to embed theoretical knowledge for globally accurate predictions. The proposed method is compared with other prediction and compensation approaches for four robotic motion tasks. It significantly reduces the time consumption by approximately 89.3% compared with direct measurements and achieves a maximum reduction of approximately 90% in robot pose errors. This substantial enhancement in pose accuracy promotes high-quality applications of robots in the industrial sector.

Real-time prediction of deposited bead width in L-DED using semi-supervised transfer learning

Real-time prediction of deposited bead width in L-DED using semi-supervised transfer learning

Semi-supervised transfer learning with hierarchical self-regularization

Both semi-supervised learning and transfer learning aim at lowering the annotation burden for training models. However, such two tasks are usually studied separately, i.e. most semi-supervised learning algorithms train models from scratch while transfer learning assumes pre-trained models as the initialization. In this work, we focus on a previously-less-concerned setting that further reduces the annotation efforts through incorporating both semi-supervised and transfer learning, where specifically a pre-trained source model is used as the initialization of semi-supervised learning. As those powerful pre-trained models are ubiquitously available nowadays and can considerably benefit various down-streaming tasks, such a setting is relevant to real-world applications yet challenging to design effective algorithms.Aiming at enabling transfer learning under semi-supervised settings, we propose a hierarchical self-regularization mechanism to exploit unlabeled samples more effectively, where a novel self-regularizer has been introduced to incorporate both individual-level and population-level regularization terms. The former term employs self-distillation to regularize learned deep features for each individual sample, and the latter one enforces self-consistency on feature distributions between labeled and unlabeled samples. Samples involved in both regularizers are weighted by an adaptive strategy, where self-regularization effects of both terms are adaptively controlled by the confidence of every sample. To validate our algorithm, exhaustive experiments have been conducted on diverse datasets such as CIFAR-10 for general object recognition, CUB-200-2011/MIT-indoor-67 for fine-grained classification and MURA for medical image classification. Compared with state-of-the-art semi-supervised learning methods including Pseudo Label, Mean Teacher, MixMatch and FixMatch, our algorithm demonstrates two advantages: first of all, the proposed approach adopts a new point of view to tackle problems caused by inadequate supervision and achieves very competitive results; then, it is complementary to these state-of-the-art methods and thus can be combined with them to get additional improvements. Furthermore, our method can also be applied to fully supervised transfer learning and self-supervised learning. We have published our code at https://github.com/SHI-Labs/Semi-Supervised-Transfer-Learning.

Semi-supervised transfer learning-based automatic weld defect detection and visual inspection

Welding is the most used fusing technology in the fabrication process of engineering structures including buildings, industrial facilities, bridges, and vital pipelines. Regular inspection of these welds is important for their overall safety and reliable usage. The need for fully automated, highly efficient structural health monitoring has encouraged researchers to invest their focus on new technologies involving artificial intelligence and computer vision. In this paper, a novel Semi-supervised Transfer learning based Multi-domain learning (ST-MDL) network is proposed for automatic weld image segmentation. Six traditional segmentation models based on U-net architecture have been utilized for comparative analysis with the proposed ST-MDL approach. The best performing model is employed for semantic segmentation of weld images to identify and measure the weld porosity inside a weld image. A novel end-to-end fully automated weld visual inspection framework (W-VIF) has also been proposed, where the measurements of segmented weld images are compared with the American welding society handbook of visual inspection standards to fully automate the process of visual inspection. The results demonstrate that the trained models are capable of identifying and quantifying weld defects, and hence a practical fully automated weld visual inspection procedure is created.

Research and Application of Semi-Supervised Category Dictionary Model Based on Transfer Learning

Classification tasks are pivotal across diverse applications, yet the burgeoning amount of data, coupled with complicating factors such as noise, exacerbates the challenge of classifying complex data. For algorithms that require a large amount of data, the annotation work for datasets is also exceptionally complex and tedious. Drawing upon existing research, this paper first introduces a novel semi-supervised category dictionary model based on transfer learning (SSDT). This model is designed to construct a more representative category dictionary and to delineate the associations of information across different domains, utilizing the lens of conditional probability distribution. This approach is particularly apt for semi-supervised transfer learning scenarios. Subsequently, the proposed method is applied to the domain of bearing fault diagnosis. This model is suitable for transfer scenarios; moreover, its semi-supervised characteristic eliminates the need for labeling the entire input dataset, significantly reducing manual workload. Experimental results attest to the model’s practical utility. When benchmarked against other 6 models, the SSDT model demonstrates enhanced generalization performance.

Cross-hospital Sepsis Early Detection via Semi-supervised Optimal Transport with Self-paced Ensemble.

Leveraging machine learning techniques for Sepsis early detection and diagnosis has attracted increasing interest in recent years. However, most existing methods require a large amount of labeled training data, which may not be available for a target hospital that deploys a new Sepsis detection system. More seriously, as treated patients are diversified between hospitals, directly applying a model trained on other hospitals may not achieve good performance for the target hospital. To address this issue, we propose a novel semi-supervised transfer learning framework based on optimal transport theory and self-paced ensemble for Sepsis early detection, called SPSSOT, which can efficiently transfer knowledge from the source hospital (with rich labeled data) to the target hospital (with scarce labeled data). Specifically, SPSSOT incorporates a new optimal transport-based semi-supervised domain adaptation component that can effectively exploit all the unlabeled data in the target hospital. Moreover, self-paced ensemble is adapted in SPSSOT to alleviate the class imbalance issue during transfer learning. In a nutshell, SPSSOT is an end-to-end transfer learning method that automatically selects suitable samples from two domains (hospitals) respectively and aligns their feature spaces. Extensive experiments on two open clinical datasets, MIMIC-III and Challenge, demonstrate that SPSSOT outperforms state-of-the-art transfer learning methods by improving 1-3% of AUC.

A novel compound fault diagnosis method for rolling bearings based on graph label manifold metric transfer

Metric transfer learning overcomes the problem of no domain distribution alienation in traditional sample transfer, and also improves the classification accuracy. However, some problems exist; for example, the target domain must have labelled samples without considering the manifold connection between data samples, and the algorithm is not robust under sparse samples. A compound fault diagnosis method for rolling bearings based on graph label manifold metric transfer is proposed. First, the pseudo-label of the target domain is acquired through a weighted graph constructed on the source domain with known fault labels. In order to ensure the accuracy of metrics in manifold space, we adopt the method of graph label propagation (LP) to gradually spread the label information. Then, the sample data are mapped to the low-dimensional manifold space by locality preserving projections (LPP), which gives higher weight to the source domain samples close to the target domain to solve the domain offset problem. At the same time, combined with the labelled samples of the target domain in LP, the manifold distance measure is learned to minimize the intra-class distance and maximize the class spacing, eliminate the overlap phenomenon in the domain and realize the distribution alienation between classes in the domain. After each iteration, unsupervised manifold metric transfer learning is realized by increasing the range of LP and gradually reducing the error of graph LP. Experiments show that the new unsupervised transfer learning method has better fault identification ability than the semi-supervised transfer learning method. In the case of sparse training samples, it still maintains a high diagnostic accuracy, and the highest fault recognition accuracy can reach more than 97% with good robustness.

Research on Shaft Current Damage Identification of Variable Condition Motor Bearings Based on Multiscale Feature Label Propagation and Manifold Metric Transfer

The current damage is the most stubborn and difficult fault of high-power motor bearings because its vibration characteristics are easily confused with those of ordinary bearing mechanical faults. If it is discriminated as an ordinary mechanical fault without electrical insulation protection, the current damage of bearing shafts will still repeatedly appear. Aiming at the problem that it is difficult to identify the bearing current damage fault under variable working conditions, a bearing shaft current damage identification method based on multiscale feature label propagation and manifold metric transfer (MFLP-MMT) is proposed. Firstly, the multiscale sub-band signal is obtained by wavelet packet decomposition, and the multiscale sub-band fuzzy entropy is obtained by calculating its fuzzy entropy. Then, according to the extracted features, a neighbor graph is constructed on the source domain of the known fault label to obtain the pseudo label of the target domain sample, and the source domain label information is gradually diffused by way of the graph label propagation. The multiscale sub-band fuzzy entropy of the sample is mapped to the low-dimensional manifold space by locality preserving projections (LPP), and the source domain samples close to the target domain are given higher weights by cross-domain density ratio estimation to solve the problem of domain offset. Combined with the label samples of the target domain in label propagation, the manifold distance metric is learned to minimize the intra-class distance and maximize the inter-class distance in the domain and eliminate the overlapping phenomenon in the domain. By increasing the range of label propagation after each iteration, the label propagation error of the leading graph is gradually reduced, and unsupervised metric transfer learning is realized. The experimental results show that the new method is superior to the semi-supervised transfer learning method in fault identification ability; the highest fault identification accuracy can reach 100% and it has a good robustness.

Source Free Semi-Supervised Transfer Learning for Diagnosis of Mental Disorders on fMRI Scans.

The high prevalence of mental disorders gradually poses a huge pressure on the public healthcare services. Deep learning-based computer-aided diagnosis (CAD) has emerged to relieve the tension in healthcare institutions by detecting abnormal neuroimaging-derived phenotypes. However, training deep learning models relies on sufficient annotated datasets, which can be costly and laborious. Semi-supervised learning (SSL) and transfer learning (TL) can mitigate this challenge by leveraging unlabeled data within the same institution and advantageous information from source domain, respectively. This work is the first attempt to propose an effective semi-supervised transfer learning (SSTL) framework dubbed S3TL for CAD of mental disorders on fMRI data. Within S3TL, a secure cross-domain feature alignment method is developed to generate target-related source model in SSL. Subsequently, we propose an enhanced dual-stage pseudo-labeling approach to assign pseudo-labels for unlabeled samples in target domain. Finally, an advantageous knowledge transfer method is conducted to improve the generalization capability of the target model. Comprehensive experimental results demonstrate that S3TL achieves competitive accuracies of 69.14%, 69.65%, and 72.62% on ABIDE-I, ABIDE-II, and ADHD-200 datasets, respectively. Furthermore, the simulation experiments also demonstrate the application potential of S3TL through model interpretation analysis and federated learning extension.

Differentially private transferrable deep learning with membership-mappings

Despite a recent surge of research interest in privacy and transferrable deep learning, optimizing the tradeoff between privacy requirements and performance of machine learning models remains a challenge. This motivates the development of an approach that optimizes both privacy-preservation mechanism and learning of the deep models for achieving a robust performance. This paper considers the problem of semi-supervised transfer and multi-task learning under differential privacy framework. An alternative conception of deep autoencoder, referred to as Conditionally Deep Membership-Mapping Autoencoder (CDMMA), is considered for transferrable deep learning. Under practice-oriented settings, an analytical solution for the learning of CDMMA can be derived by means of variational optimization. The paper proposes a transfer and multi-task learning approach that combines CDMMA with a tailored noise adding mechanism to transfer knowledge from source to target domain in a privacy-preserving manner.

Semi-Supervised Transfer Learning Methodology for Fault Detection and Diagnosis in Air-Handling Units

Heating, ventilation and air-conditioning (HVAC) systems are the major energy consumers among buildings’ equipment. Reliable fault detection and diagnosis schemes can effectively reduce their energy consumption and maintenance costs. In this respect, data-driven approaches have shown impressive results, but their accuracy depends on the availability of representative data to train the models, which is not common in real applications. For this reason, transfer learning is attracting growing attention since it tackles the problem by leveraging the knowledge between datasets, increasing the representativeness of fault scenarios. However, to date, research on transfer learning for heating, ventilation and air-conditioning has mostly been focused on learning algorithmic, overlooking the importance of a proper domain similarity analysis over the available data. Thus, this study proposes the design of a transfer learning approach based on a specific data selection methodology to tackle dissimilarity issues. The procedure is supported by neural network models and the analysis of eventual prediction uncertainties resulting from the assessment of the target application samples. To verify the proposed methodology, it is applied to a semi-supervised transfer learning case study composed of two publicly available air-handling unit datasets containing some fault scenarios. Results emphasize the potential of the proposed domain dissimilarity analysis reaching a classification accuracy of 92% under a transfer learning framework, an increase of 37% in comparison to classical approaches.

Bayesian transfer learning with active querying for intelligent cross-machine fault prognosis under limited data

Most existing deep learning (DL)-based health prognostic methods assume that the training and testing datasets are from identical machines operating under similar conditions requiring massive labelled condition monitoring (CM) data to guarantee the prediction accuracy and generalisation capacity. However, these strict restrictions significantly hinder the deployment of the dl-based prognostic methods in real industries. In this paper, a Bayesian semi-supervised transfer learning with active querying-based intelligent fault prognostic framework is developed for remaining useful life (RUL) prediction across completely different machines under limited data. The proposed method strategically integrates the advantages of transfer learning (TL) and active learning in the Bayesian deep learning (BDL) framework. In the proposed framework, Bayesian neural networks with Monte Carlo dropout inference are utilised to quantify RUL prediction uncertainty, which is further leveraged to develop an active querying-based training data selection mechanism. Moreover, TL is simultaneously embedded into the BDL framework to relieve data distribution discrepancies existing among the completely different machines. The experimental verifications from open-sourced bearing datasets and lab testing-based lithium-ion battery degradation datasets demonstrate that the proposed framework can effectively and reliably achieve bi-directional transfer fault prognostic tasks under limited labelled CM data in target domain. Finally, generalisation and superiority of the proposed method are also validated by comparing with other state-of-the-art methods.

Intra-Inter Subject Self-Supervised Learning for Multivariate Cardiac Signals

Learning information-rich and generalizable representations effectively from unlabeled multivariate cardiac signals to identify abnormal heart rhythms (cardiac arrhythmias) is valuable in real-world clinical settings but often challenging due to its complex temporal dynamics. Cardiac arrhythmias can vary significantly in temporal patterns even for the same patient (i.e., intra subject difference). Meanwhile, the same type of cardiac arrhythmia can show different temporal patterns among different patients due to different cardiac structures (i.e., inter subject difference). In this paper, we address the challenges by proposing an Intra-Inter Subject Self-Supervised Learning (ISL) model that is customized for multivariate cardiac signals. Our proposed ISL model integrates medical knowledge into self-supervision to effectively learn from intra-inter subject differences. In intra subject self-supervision, ISL model first extracts heartbeat-level features from each subject using a channel-wise attentional CNN-RNN encoder. Then a stationarity test module is employed to capture the temporal dependencies between heartbeats. In inter subject self-supervision, we design a set of data augmentations according to the clinical characteristics of cardiac signals and perform contrastive learning among subjects to learn distinctive representations for various types of patients. Extensive experiments on three real-world datasets were conducted. In a semi-supervised transfer learning scenario, our pre-trained ISL model leads about 10% improvement over supervised training when only 1% labeled data is available, suggesting strong generalizability and robustness of the model.

Semi-Supervised Transfer Learning Method for Bearing Fault Diagnosis with Imbalanced Data

Fault diagnosis is essential for assuring the safety and dependability of rotating machinery systems. Several emerging techniques, especially artificial intelligence-based technologies, are used to overcome the difficulties in this field. In most engineering scenarios, machines perform in normal conditions, which implies that fault data may be hard to acquire and limited. Therefore, the data imbalance and the deficiency of labels are practical challenges in the fault diagnosis of machinery bearings. Among the mainstream methods, transfer learning-based fault diagnosis is highly effective, as it transfers the results of previous studies and integrates existing resources. The knowledge from the source domain is transferred via Domain Adversarial Training of Neural Networks (DANN) while the dataset of the target domain is partially labeled. A semi-supervised framework based on uncertainty-aware pseudo-label selection (UPS) is adopted in parallel to improve the model performance by utilizing abundant unlabeled data. Through experiments on two bearing datasets, the accuracy of bearing fault classification surpassed the independent approaches.

Fault Diagnosis for Power Transformers through Semi-Supervised Transfer Learning.

The fault diagnosis of power transformers is a challenging problem. The massive multisource fault is heterogeneous, the type of fault is undetermined sometimes, and one device has only met a few kinds of faults in the past. We propose a fault diagnosis method based on deep neural networks and a semi-supervised transfer learning framework called adaptive reinforcement (AR) to solve the above limitations. The innovation of this framework consists of its enhancement of the consistency regularization algorithm. The experiments were conducted on real-world 110 kV power transformers’ three-phase fault grounding currents of the iron cores from various devices with four types of faults: Phases A, B, C and ABC to ground. We trained the model on the source domain and then transferred the model to the target domain, which included the unbalanced and undefined fault datasets. The results show that our proposed model reaches over accuracy in classifying the type of fault and outperforms other popular networks. Our AR framework fits target devices’ fault data with fewer dozen epochs than other novel semi-supervised techniques. Combining the deep neural network and the AR framework helps diagnose the power transformers, which lack diagnosis knowledge, with much less training time and reliable accuracy.

Transfer Learning-Based Autosegmentation of Primary Tumor Volumes of Glioblastomas Using Preoperative MRI for Radiotherapy Treatment.

ObjectivesGlioblastoma is the most common primary malignant brain tumor in adults and can be treated with radiation therapy. However, tumor target contouring for head radiation therapy is labor-intensive and highly dependent on the experience of the radiation oncologist. Recently, autosegmentation of the tumor target has been playing an increasingly important role in the development of radiotherapy plans. Therefore, we established a deep learning model and improved its performance in autosegmenting and contouring the primary gross tumor volume (GTV) of glioblastomas through transfer learning.MethodsThe preoperative MRI data of 20 patients with glioblastomas were collected from our department (ST) and split into a training set and testing set. We fine-tuned a deep learning model for autosegmentation of the hippocampus on separate MRI scans (RZ) through transfer learning and trained this deep learning model directly using the training set. Finally, we evaluated the performance of both trained models in autosegmenting glioblastomas using the testing set.ResultsThe fine-tuned model converged within 20 epochs, compared to over 50 epochs for the model trained directly by the same training set, and demonstrated better autosegmentation performance [Dice similarity coefficient (DSC) 0.9404 ± 0.0117, 95% Hausdorff distance (95HD) 1.8107 mm ±0.3964mm, average surface distance (ASD) 0.6003 mm ±0.1287mm] than the model trained directly (DSC 0.9158±0.0178, 95HD 2.5761 mm ± 0.5365mm, ASD 0.7579 mm ± 0.1468mm) with the same test set. The DSC, 95HD, and ASD values of the two models were significantly different (P<0.05).ConclusionA model developed with semisupervised transfer learning and trained on independent data achieved good performance in autosegmenting glioblastoma. The autosegmented volume of glioblastomas is sufficiently accurate for radiotherapy treatment, which could have a positive impact on tumor control and patient survival.