Deep Learning-based Domain Adaptation Research Articles

Monitoring of Laser Powder Bed Fusion process by bridging dissimilar process maps using deep learning-based domain adaptation on acoustic emissions

Advances in sensorization and identification of information embedded inside sensor signatures during manufacturing processes using Machine Learning (ML) algorithms for better decision-making have become critical enablers in building data-driven monitoring systems. In the Laser Powder Bed Fusion (LPBF) process, data-driven-based process monitoring is gaining popularity since it allows for real-time component quality verification. Real-time qualification of the additively manufactured parts has a significant advantage as the cost of conventional post-manufacturing inspection methods can be reduced. Also, corrective actions or build termination could be done to save machine time and resources. However, despite the successful development in addressing monitoring needs in LPBF processes, less research has been paid to the ML model's robustness in decision-making when dealing with variations in data distribution from the laser-material interaction owing to different process spaces. Inspired by the idea of domain adaptation in ML, in this work, we propose a deep learning-based unsupervised domain adaptation technique to tackle shifts in data distribution owing to different process parameter spaces. The temporal waveforms of acoustic emissions from the LPBF process zone corresponding to three regimes, namely Lack of Fusion, conduction, and keyhole, were acquired on two different 316 L stainless steel powder distributions (> 45 µm and < 45 µm) with two different parameter sets. Temporal and spectral analysis of the acoustic waveforms corresponding to the powder distributions treated with different laser parameters showed the presence of offset in the data distribution, which was subsequently treated with the proposed unsupervised domain adaptation technique to have an ML model that could be generalized. Furthermore, the prediction accuracy of the proposed methodology between the two distributions showed the feasibility of adapting to the newer environment unsupervisedly and improving the ML model's generalizability.

Deep learning-based domain adaptation for a generalized detection of wear phenomena during blanking

Blanking plays an important role in most manufacturing chains as this forming operation defines the final geometry as well as functional properties of the product. As a result of high production rates combined with excessive mechanical and dynamic loads, wear is an unavoidable phenomenon in such processes and directly worsens their economic efficiency. Although wear plays this crucial role, its detection is mainly limited to the manual control or inline binary fault detection. In this context, data-driven approaches such as conventional machine or deep learning allow to overcome this challenge by an inline wear state estimation. Unfortunately, their application in practice is limited since changes to the system configuration significantly degrade the performance of the implemented data-driven model. To address this problem, an adapted deep learning-based domain adaptation (DA) technique is proposed in this paper. With this new procedure, sensorial acquired force singles are combined with a deep network architecture to enable an inline detection of finely graded wear states in a blanking process, even if there has been a change in system configuration. As a result, high accuracies for classifying the 11 wear states are achieved without retraining the underlying models, in case of a distribution discrepancy between source and target domain. In contrast to no DA technique as well as a conventional DA technique applied to transfer tasks, the proposed method improves the classification accuracy in the target domain up to 95%. Validation of this approach based on different real-world data sets demonstrates its suitability for a generalized inline wear state estimation, even under varying process boundaries.

Cross-spatiotemporal land-cover classification from VHR remote sensing images with deep learning based domain adaptation

Automatic land use/land cover (LULC) classification from very high resolution (VHR) remote sensing images can provide us with rapid, large-scale, and fine-grained understanding of the urbanization and ecosystem processes on the Earth’s surface. Although the burgeoning use of deep learning technology has boosted land cover classification from VHR images, one of the key challenges has not been investigated in depth, i.e., the current data-driven deep learning models are heavily reliant on the high similarity between the distributions of the labeled training data, i.e., the source data, and the unlabeled data, i.e., the target data. However, in practice, this condition is rarely met, and preparing labels manually every time for the target data is unrealistic. In this paper, we comprehensively evaluate the domain adaptation methods for modern deep learning based classification models. Domain adaptation is aimed at narrowing the domain shift between the labeled source data and the unlabeled target data, providing a practical way for a deep learning based model to fully utilize historical training data and get rid of the need for continual manual work. Furthermore, we propose a novel two-stage Domain Adaptation method for Cross-Spatio-Temporal classification called the DACST method with the inputs of labeled source data and the unlabeled target data. It consists of an image-level adaptation stage that aligns the appearance of the source and target data and produces the target-stylized source images, and a feature-level adaptation stage that further narrows the domain shift in the deep feature space. DACST significantly improves the spatiotemporal transferability of the classification model, which is embedded in the feature-level adaptation stage, to output a satisfactory classification map. In this study, we also conducted a comprehensive performance evaluation of the conventional and deep learning based image-level and feature-level domain adaptation methods for VHR LULC classification. Both binary and multi-class classification was conducted in cross-temporal and cross-spatiotemporal scenes in five large-scale datasets from around the world. The very high performance and the best robustness of the proposed method suggests that a new baseline of cross-domain VHR land cover classification in the deep learning age is being witnessed. The experiments also indicate that both the conventional and deep learning based image-level domain adaptation methods function in various situations, but almost all the feature-level methods are highly unstable on different data.

Deep learning based domain adaptation for mitochondria segmentation on EM volumes

Background and Objective: Accurate segmentation of electron microscopy (EM) volumes of the brain is essential to characterize neuronal structures at a cell or organelle level. While supervised deep learning methods have led to major breakthroughs in that direction during the past years, they usually require large amounts of annotated data to be trained, and perform poorly on other data acquired under similar experimental and imaging conditions. This is a problem known as domain adaptation, since models that learned from a sample distribution (or source domain) struggle to maintain their performance on samples extracted from a different distribution or target domain. In this work, we address the complex case of deep learning based domain adaptation for mitochondria segmentation across EM datasets from different tissues and species. Methods: We present three unsupervised domain adaptation strategies to improve mitochondria segmentation in the target domain based on (1) state-of-the-art style transfer between images of both domains; (2) self-supervised learning to pre-train a model using unlabeled source and target images, and then fine-tune it only with the source labels; and (3) multi-task neural network architectures trained end-to-end with both labeled and unlabeled images. Additionally, to ensure good generalization in our models, we propose a new training stopping criterion based on morphological priors obtained exclusively in the source domain. The code and its documentation are publicly available at https://github.com/danifranco/EM_domain_adaptation. Results: We carried out all possible cross-dataset experiments using three publicly available EM datasets. We evaluated our proposed strategies and those of others based on the mitochondria semantic labels predicted on the target datasets. Conclusions: The methods introduced here outperform the baseline methods and compare favorably to the state of the art. In the absence of validation labels, monitoring our proposed morphology-based metric is an intuitive and effective way to stop the training process and select in average optimal models.

Intelligent Robust Cross-Domain Fault Diagnostic Method for Rotating Machines Using Noisy Condition Labels

Cross-domain fault diagnosis methods have been successfully and widely developed in the past years, which focus on practical industrial scenarios with training and testing data from numerous machinery working regimes. Due to the remarkable effectiveness in such problems, deep learning-based domain adaptation approaches have been attracting increasing attention. However, the existing methods in the literature are generally lower compared to environmental noise and data availability, and it is difficult to achieve promising performance under harsh practical conditions. This paper proposes a new cross-domain fault diagnosis method with enhanced robustness. Noisy labels are introduced to significantly increase the generalization ability of the data-driven model. Promising diagnosis performance can be obtained with strong noise interference in testing, as well as in practical cases with low-quality data. Experiments on two rotating machinery datasets are carried out for validation. The results indicate that the proposed algorithm is well suited to be applied in real industrial environments to achieve promising performance with variations of working conditions.

Single-channel EEG based insomnia detection with domain adaptation

Insomnia is one of the most common sleep disorders which can dramatically impair life quality and negatively affect an individual's physical and mental health. Recently, various deep learning based methods have been proposed for automatic and objective insomnia detection, owing to the great success of deep learning techniques. However, due to the scarcity of public insomnia data, a deep learning model trained on a dataset with a small number of insomnia subjects may compromise the generalization capacity of the model and eventually limit the performance of insomnia detection. Meanwhile, there have been a number of public EEG datasets collected from a large number of healthy subjects for various sleep research tasks such as sleep staging. Therefore, to utilize such abundant EEG datasets for addressing the data scarcity issue in insomnia detection, in this paper we propose a domain adaptation based model to better extract insomnia related features of the target domain by leveraging stage annotations from the source domain. For each domain, two pairs of common encoder and private encoder are firstly trained to extract sleep related features and sleep irrelevant features, respectively. In order to further discriminate source domain and target domain, a domain classifier is introduced. Then, the common encoder of the target domain will be used together with the Long Short Term Memory (LSTM) network for insomnia detection. To the best of our knowledge, this is the first deep learning based domain adaptation model using single channel raw EEG signals to detect insomnia at subject level. We use the Montreal Archive of Sleep Studies (MASS) dataset which contains only healthy subjects as source domain and two datasets which contain both healthy and insomnia subjects as target domain to validate our model's generalizability. Experimental results on the two target domain datasets (a public one and an in-house one) demonstrate that our model generalizes well on two target domain datasets with different sampling rates. In particular, our proposed method is able to improve insomnia detection performance from 50.0% to 90.9% and 66.7%–79.2% in terms of accuracy on the two target domain datasets, respectively.

DBAM: Making Virtual Metrology/Soft sensing with time series data scalable through Deep Learning

Machine Learning (ML) based technologies, like Virtual Metrology (VM)/Soft Sensing, Predictive Maintenance and Fault Detection, have been successfully applied in the past recent years in data intensive manufacturing industries, like semiconductor manufacturing, to improve process monitoring and related operations. Standardization and alignment over multiple equipment is a key element to ensure industry-wide adoption and scalability for ML-based technologies in complex production environment. In this work we address the topic of VM/Soft Sensing – a particular ML-based technology for process control – in the context of equipment matching and scalability. We present a Deep Learning-based domain adaptation approach, called DANN-Based Model Alignment (DBAM), that provides a common VM model for two identical-in-design systems whose data are following different distributions. The proposed approach has the merit of (i) exploiting directly raw sensor data (that typically present themselves in the form of time series) and (ii) offering interpretability of the features. The proposed approach is compared against other approaches in the literature for VM/Soft Sensing on a real-world case study from semiconductor manufacturing.

A multi-representation-based domain adaptation network for fault diagnosis

Deep learning-based domain adaptation algorithms with various representations have been recently developed to address the domain shift problem in mechanical fault diagnosis. However, few research have focused on potential improvements through multiple representations. Thus, a multi-representation-based domain adaptation network is proposed in this paper. Three complementary time–frequency representations are first proposed to serve as input-based multiple representations for the subsequent parallel models. Then, parallel models with improved inception modules are trained to obtain feature-based multiple domain-invariant representations. Finally, ensemble learning through majority voting is used to obtain the final results. Comprehensive experimental results on two test rigs reveal that the proposed algorithm outperforms state-of-the-art single-representation-based domain adaptation algorithms in terms of cross-domain fault diagnosis. Furthermore, visualization results demonstrate that the proposed algorithm extracts transferable features and takes advantage of ensemble learning to achieve high-precision diagnosis.

Domain adaptation of object detector using scissor-like networks

When the training data and the test data do not obey the same distribution, the performances of many object detection methods always decrease greatly. Naturally, domain adaptation methods at feature level are proposed. The basic idea is to adapt the feature extraction network such that the feature distributions of the source and target domains match. We propose a new method that is built directly on the Faster R-CNN model, which not only aligns the source and target data features, but also forces their generated features closer together to further align the source and target domains. Moreover, compared with previous approaches, we construct a more powerful discriminator and a simple generator to solve the domain adaptation problem. The model works like a pair of scissors, so we call it Scissors Networks (SN). We conduct extensive experiments on popular datasets, including Cityscapes, Foggy Cityscapes, SIM10k and KITTI. The experimental results demonstrate that our algorithm is superior to the state-of-the-art deep learning based domain adaptation approaches.

Deep Domain Adaptation Based Multi-Spectral Salient Object Detection

Salient Object Detection (SOD) plays an important role in many image-related multimedia applications. Although there are many existing research works about the salient object detection in traditional RGB (visible-light spectrum) images, there are still many complex situations that regular RGB images cannot provide enough cues for the accurate SOD, such as the shadow effect, similar appearance between background and foreground, strong or insufficient illumination, etc. Because of the success of near-infrared spectrum in many computer vision tasks, we explore the multi-spectral SOD in the synchronized RGB images and near-infrared (NIR) images for the both simple and complex situations. We assume that the RGB SOD in the existing RGB image datasets could provide references for the multi-spectral SOD problem. In this paper, we mainly model this research problem as a deep learning based domain adaptation from the traditional RGB image data (source domain) to the multi-spectral data (target domain), and an adversarial deep domain adaptation model is proposed. We first collect and will publicize a large multi-spectral dataset, RGBN-SOD dataset, including 780 synchronized RGB and NIR image pairs for the multi-spectral SOD problem in the simple and complex situations. Intensive experimental results show the effectiveness and accuracy of the proposed deep domain adaptation for the multi-spectral SOD. Besides, due to the absence of research on the field of multi-spectral co-saliency detection, we also collect 200 synchronized RGB and NIR image pairs in addition to explore the multi-spectral co-saliency detection.

DOMAIN ADAPTATION WITH CYCLEGAN FOR CHANGE DETECTION IN THE AMAZON FOREST

Abstract. Deep learning classification models require large amounts of labeled training data to perform properly, but the production of reference data for most Earth observation applications is a labor intensive, costly process. In that sense, transfer learning is an option to mitigate the demand for labeled data. In many remote sensing applications, however, the accuracy of a deep learning-based classification model trained with a specific dataset drops significantly when it is tested on a different dataset, even after fine-tuning. In general, this behavior can be credited to the domain shift phenomenon. In remote sensing applications, domain shift can be associated with changes in the environmental conditions during the acquisition of new data, variations of objects’ appearances, geographical variability and different sensor properties, among other aspects. In recent years, deep learning-based domain adaptation techniques have been used to alleviate the domain shift problem. Recent improvements in domain adaptation technology rely on techniques based on Generative Adversarial Networks (GANs), such as the Cycle-Consistent Generative Adversarial Network (CycleGAN), which adapts images across different domains by learning nonlinear mapping functions between the domains. In this work, we exploit the CycleGAN approach for domain adaptation in a particular change detection application, namely, deforestation detection in the Amazon forest. Experimental results indicate that the proposed approach is capable of alleviating the effects associated with domain shift in the context of the target application.

Multi-source domain adaptation for quality control in retail food packaging

Retail food packaging contains information which informs choice and can be vital to consumer health, including product name, ingredients list, nutritional information, allergens, preparation guidelines, pack weight, storage and shelf life information (use-by/best before dates). The presence and accuracy of such information is critical to ensure a detailed understanding of the product and to reduce the potential for health risks. Consequently, erroneous or illegible labeling has the potential to be highly detrimental to consumers and many other stakeholders in the supply chain. In practice, due to the high volume of food packages that go through the supply chain, mistakes do occur therefore good quality of images are needed to verify the correctness of the information. In this paper, a multi-source deep learning-based domain adaptation system is proposed and tested to identify and verify the presence and legibility of use-by date information from food packaging photos taken as part of the validation process as the products pass along the food production line. This was achieved by improving the generalization of the techniques via incorporating new loss functions and making use of multi-source datasets in order to extract domain-invariant representations for all domains and aligning distribution of all pairs of source and target domains in a common feature space, along with the class boundaries. The proposed system performed very well in the conducted experiments, for automating the verification process and reducing labeling errors that could otherwise threaten public health and contravene legal requirements for food packaging information and accuracy. Comprehensive experiments on our food packaging datasets demonstrate that the proposed multi-source deep domain adaptation method significantly improves the classification accuracy and therefore has great potential for application and beneficial impact in food manufacturing control systems.

Advancing Medical Imaging Informatics by Deep Learning-Based Domain Adaptation.

Summary Introduction : There has been a rapid development of deep learning (DL) models for medical imaging. However, DL requires a large labeled dataset for training the models. Getting large-scale labeled data remains a challenge, and multi-center datasets suffer from heterogeneity due to patient diversity and varying imaging protocols. Domain adaptation (DA) has been developed to transfer the knowledge from a labeled data domain to a related but unlabeled domain in either image space or feature space. DA is a type of transfer learning (TL) that can improve the performance of models when applied to multiple different datasets. Objective : In this survey, we review the state-of-the-art DL-based DA methods for medical imaging. We aim to summarize recent advances, highlighting the motivation, challenges, and opportunities, and to discuss promising directions for future work in DA for medical imaging. Methods : We surveyed peer-reviewed publications from leading biomedical journals and conferences between 2017-2020, that reported the use of DA in medical imaging applications, grouping them by methodology, image modality, and learning scenarios. Results : We mainly focused on pathology and radiology as application areas. Among various DA approaches, we discussed domain transformation (DT) and latent feature-space transformation (LFST). We highlighted the role of unsupervised DA in image segmentation and described opportunities for future development. Conclusion : DA has emerged as a promising solution to deal with the lack of annotated training data. Using adversarial techniques, unsupervised DA has achieved good performance, especially for segmentation tasks. Opportunities include domain transferability, multi-modal DA, and applications that benefit from synthetic data.

Generative Adversarial Domain Adaptation for Nucleus Quantification in Images of Tissue Immunohistochemically Stained for Ki-67.

PURPOSEWe focus on the problem of scarcity of annotated training data for nucleus recognition in Ki-67 immunohistochemistry (IHC)–stained pancreatic neuroendocrine tumor (NET) images. We hypothesize that deep learning–based domain adaptation is helpful for nucleus recognition when image annotations are unavailable in target data sets.METHODSWe considered 2 different institutional pancreatic NET data sets: one (ie, source) containing 38 cases with 114 annotated images and the other (ie, target) containing 72 cases with 20 annotated images. The gold standards were manually annotated by 1 pathologist. We developed a novel deep learning–based domain adaptation framework to count different types of nuclei (ie, immunopositive tumor, immunonegative tumor, nontumor nuclei). We compared the proposed method with several recent fully supervised deep learning models, such as fully convolutional network-8s (FCN-8s), U-Net, fully convolutional regression network (FCRN) A, FCRNB, and fully residual convolutional network (FRCN). We also evaluated the proposed method by learning with a mixture of converted source images and real target annotations.RESULTSOur method achieved an F1 score of 81.3% and 62.3% for nucleus detection and classification in the target data set, respectively. Our method outperformed FCN-8s (53.6% and 43.6% for nucleus detection and classification, respectively), U-Net (61.1% and 47.6%), FCRNA (63.4% and 55.8%), and FCRNB (68.2% and 60.6%) in terms of F1 score and was competitive with FRCN (81.7% and 70.7%). In addition, learning with a mixture of converted source images and only a small set of real target labels could further boost the performance.CONCLUSIONThis study demonstrates that deep learning–based domain adaptation is helpful for nucleus recognition in Ki-67 IHC stained images when target data annotations are not available. It would improve the applicability of deep learning models designed for downstream supervised learning tasks on different data sets.

Partial transfer learning in machinery cross-domain fault diagnostics using class-weighted adversarial networks

Recently, transfer learning has been receiving growing interests in machinery fault diagnosis due to its strong generalization across different industrial scenarios. The existing methods generally assume identical label spaces, and propose minimizing marginal distribution discrepancy between source and target domains. However, this assumption usually does not hold in real industries, where testing data mostly contain a subspace of the source label space. Therefore, transferring diagnosis knowledge from a comprehensive source domain to a target domain with limited machine conditions is motivated. This challenging partial transfer learning problem is addressed in this study using deep learning-based domain adaptation method. A class weighted adversarial neural network is proposed to encourage positive transfer of the shared classes and ignore the source outliers. Experimental results on two rotating machinery datasets suggest the proposed method is promising for partial transfer learning.

Deep Learning-Based Domain Adaptation Method for Fault Diagnosis in Semiconductor Manufacturing

Quality inspection in semiconductor manufacturing is of great importance in the modern industries. In the recent years, intelligent data-driven condition monitoring methods have been successfully developed and applied in the industrial applications. However, despite the promising condition monitoring performance, the existing methods generally assume the training and testing data are from the same distribution. In practice, due to variations in manufacturing process, the collected data are usually subject to different distributions in different operating conditions, that significantly deteriorates the performance of the data-driven methods. To address this issue, this paper proposes a deep learning-based domain adaptation method for fault diagnosis in semiconductor manufacturing. The maximum mean discrepancy metric is optimized on the learned high-level data representation in the deep neural network. Experimental results on a real-world semiconductor manufacturing dataset suggest the proposed method offers an effective and generalized data-driven fault diagnosis approach for quality inspection.

Intelligent ball screw fault diagnosis using a deep domain adaptation methodology

Intelligent data-driven fault diagnosis methods have been successfully developed in the recent years. However, as one of the most important machines in the industries, the ball screw health monitoring problem has received less attention, due to the complex operating patterns and sophisticated mechanical structures. In practice, the working conditions of the ball screws usually change, that further makes the fault diagnosis problem more challenging since the data distributions are not the same. In order to address this issue, a deep learning-based domain adaptation method is proposed for the cross-domain ball screw fault diagnosis problem. The deep convolutional neural network is adopted for feature extraction and health condition classification. The maximum mean discrepancy metric is proposed to measure and optimize the data distributions of different operating conditions. A data segmentation method which is specially designed for the ball screw is further integrated. The experiments on the real ball screw condition monitoring data are carried out for validation. The results indicate the proposed approach is promising for the cross-domain diagnostic tasks of the ball screw in the real industries.

Deep learning-based cross-domain adaptation for gearbox fault diagnosis under variable speed conditions

Existing intelligent gearbox fault diagnosis approaches have two shortcomings: (a) their performance is mostly confined to manual handcrafted features, and (b) they follow a general assumption that the distribution of the data in the source domain (labeled data on which the model is trained) is similar to the target domain (unlabeled data on which the model is tested), which might not be the case in real-world applications. Substantial human expertise and domain knowledge is required for manual feature extraction, and moreover, deploying the same model for a target domain whose distribution is different from the source domain would lead to poor generalization. Since deep learning methods can automatically learn high dimensional feature representations from raw measurement data, this paper proposes a novel deep learning-based domain adaptation (DA) method for gearbox fault diagnosis under significant speed variations. A deep convolutional neural network is used as the main architecture. The paper proposes to minimize the summation of cross-entropy loss (between the labeled source domain data) and maximum mean discrepancy loss (between the labeled source and unlabeled target datasets) simultaneously to adapt the source domain model to be applied in the target domain. The proposed deep learning DA approach is evaluated using experimental data from a gearbox under variable speeds and multiple health conditions. An appropriate benchmarking with both traditional machine learning methods and other DA methods demonstrate the superiority of the proposed method.