Domain Adaptation Strategy Research Articles

Deep learning framework for congestion detection at public places via learning from synthetic data

Congestion in public places is one of the major problems in public transportation systems and causes a high level of discomfort for the commuters. Traditionally, overcrowding is detected by manually monitoring and analyzing the video streams from the surveillance cameras, which might lead to errors due to limited human activity. On the other hand, current machine learning models for automatic congestion detection require a massive amount of labeled data to train the network. These models suffer from the over-fitting problem and cannot be generalized to novel scenes. First, we propose a novel synthetic dataset for congestion detection in public places to address these problems. Secondly, we propose a Bidirectional Long-short-term-memory (Bi-LSTM) model that exploits synthetic datasets to boost the performance of congestion detection in the wild. We adopt a domain adaptation strategy to bridge the gap between the real and synthetic data by pre-train the model on the synthetic dataset and then fine-tuning the model on real data. From experiment results, we observe that the proposed framework achieves a significant performance boost on the real datasets after training on the synthetic dataset.

A novel single robot image shadow detection method based on convolutional block attention module and unsupervised learning network.

Shadow detection plays a very important role in image processing. Although many algorithms have been proposed in different environments, it is still a challenging task to detect shadows in natural scenes. In this paper, we propose a convolutional block attention module (CBAM) and unsupervised domain adaptation adversarial learning network for single image shadow detection. The new method mainly contains three steps. Firstly, in order to reduce the data deviation between the domains, the hierarchical domain adaptation strategy is adopted to calibrate the feature distribution from low level to high level between the source domain and the target domain. Secondly, in order to enhance the soft shadow detection ability of the model, the boundary adversarial branch is proposed to obtain structured shadow boundary. Meanwhile, a CBAM is added in the model to reduce the correlation between different semantic information. Thirdly, the entropy adversarial branch is combined to further suppress the high uncertainty at the boundary of the prediction results, and it obtains the smooth and accurate shadow boundary. Finally, we conduct abundant experiments on public datasets, the RMSE has the lowest values with 9.6 and BER with 6.6 on ISTD dataset, the results show that the proposed shadow detection method has better edge structure compared with the existing deep learning detection methods.

Multilevel structure-preserved GAN for domain adaptation in intravascular ultrasound analysis.

The poor generalizability of intravascular ultrasound (IVUS) analysis methods caused by the great diversity of IVUS datasets is hopefully addressed by the domain adaptation strategy. However, existing domain adaptation models underperform in intravascular structural preservation, because of the complex pathology and low contrast in IVUS images. Losing structural information during the domain adaptation would lead to inaccurate analyses of vascular states. In this paper, we propose a Multilevel Structure-Preserved Generative Adversarial Network (MSP-GAN) for transferring IVUS domains while maintaining intravascular structures. On the generator-discriminator baseline, the MSP-GAN integrates the transformer, contrastive restraint, and self-ensembling strategy, for effectively preserving structures in multi-levels, including global, local, and fine levels. For the global-level pathology maintenance, the generator explores long-range dependencies in IVUS images via an incorporated vision transformer. For the local-level anatomy consistency, a region-to-region correspondence is forced between the translated and source images via a superpixel-wise multiscale contrastive (SMC) constraint. For reducing distortions of fine-level structures, a self-ensembling mean teacher generates the pixel-wise pseudo-label and restricts the translated image via an uncertainty-aware teacher-student consistency (TSC) constraint. Experiments were conducted on 20MHz and 40MHz IVUS datasets from different medical centers. Ablation studies illustrate that each innovation contributes to intravascular structural preservation. Comparisons with representative domain adaptation models illustrate the superiority of the MSP-GAN in the structural preservation. Further comparisons with the state-of-the-art IVUS analysis accuracy demonstrate that the MSP-GAN is effective in enlarging the generalizability of diverse IVUS analysis methods and promoting accurate vessel and lumen segmentation and stenosis-related parameter quantification.

Adversarial Entropy Optimization for Unsupervised Domain Adaptation.

Domain adaptation is proposed to deal with the challenging problem where the probability distribution of the training source is different from the testing target. Recently, adversarial learning has become the dominating technique for domain adaptation. Usually, adversarial domain adaptation methods simultaneously train a feature learner and a domain discriminator to learn domain-invariant features. Accordingly, how to effectively train the domain-adversarial model to learn domain-invariant features becomes a challenge in the community. To this end, we propose in this article a novel domain adaptation scheme named adversarial entropy optimization (AEO) to address the challenge. Specifically, we minimize the entropy when samples are from the independent distributions of source domain or target domain to improve the discriminability of the model. At the same time, we maximize the entropy when features are from the combined distribution of source domain and target domain so that the domain discriminator can be confused and the transferability of representations can be promoted. This minimax regime is well matched with the core idea of adversarial learning, empowering our model with transferability as well as discriminability for domain adaptation tasks. Also, AEO is flexible and compatible with different deep networks and domain adaptation frameworks. Experiments on five data sets show that our method can achieve state-of-the-art performance across diverse domain adaptation tasks.

MSDESIS: Multitask Stereo Disparity Estimation and Surgical Instrument Segmentation.

Reconstructing the 3D geometry of the surgical site and detecting instruments within it are important tasks for surgical navigation systems and robotic surgery automation. Traditional approaches treat each problem in isolation and do not account for the intrinsic relationship between segmentation and stereo matching. In this paper, we present a learning-based framework that jointly estimates disparity and binary tool segmentation masks. The core component of our architecture is a shared feature encoder which allows strong interaction between the aforementioned tasks. Experimentally, we train two variants of our network with different capacities and explore different training schemes including both multi-task and single-task learning. Our results show that supervising the segmentation task improves our network's disparity estimation accuracy. We demonstrate a domain adaptation scheme where we supervise the segmentation task with monocular data and achieve domain adaptation of the adjacent disparity task, reducing disparity End-Point-Error and depth mean absolute error by 77.73% and 61.73% respectively compared to the pre-trained baseline model. Our best overall multi-task model, trained with both disparity and segmentation data in subsequent phases, achieves 89.15% mean Intersection-over-Union in RIS and 3.18 millimetre depth mean absolute error in SCARED test sets. Our proposed multi-task architecture is real-time, able to process ( 1280×1024 ) stereo input and simultaneously estimate disparity maps and segmentation masks at 22 frames per second. The model code and pre-trained models are made available: https://github.com/dimitrisPs/msdesis.

Attention-calibration based double-branch cross-domain person re-identification

Existing cross-domain pedestrian re-identification methods tend to utilize domain adaptation or domain generalization strategies to eliminate the differences between domains, but these methods fail to fully characterize the characteristics of unknown domain samples, resulting in poor recognition performance in unknown domain. By learning specific features and invariant features of different domains, this paper proposes a new attention-calibration double-branch cross-domain pedestrian re-identification network (ACDBNet). Firstly, the attention calibration module is utilized to extract the channel information and spatial information of an image, and locate the distinguishable local features; then, the global branch and local branch are introduced to fully learn the domain specific feature and domain invariant feature respectively, and the features in these two branches are linearly weighted fused. Experimental results on a large number of public datasets demonstrate that the proposed method has excellent performance in various environments.

Critical assessment of transformer-based AI models for German clinical notes

Healthcare data such as clinical notes are primarily recorded in an unstructured manner. If adequately translated into structured data, they can be utilized for health economics and set the groundwork for better individualized patient care. To structure clinical notes, deep-learning methods, particularly transformer-based models like Bidirectional Encoder Representations from Transformers (BERT), have recently received much attention. Currently, biomedical applications are primarily focused on the English language. While general-purpose German-language models such as GermanBERT and GottBERT have been published, adaptations for biomedical data are unavailable. This study evaluated the suitability of existing and novel transformer-based models for the German biomedical and clinical domain. We used 8 transformer-based models and pre-trained 3 new models on a newly generated biomedical corpus, and systematically compared them with each other. We annotated a new dataset of clinical notes and used it with 4 other corpora (BRONCO150, CLEF eHealth 2019 Task 1, GGPONC, and JSynCC) to perform named entity recognition (NER) and document classification tasks. General-purpose language models can be used effectively for biomedical and clinical natural language processing (NLP) tasks, still, our newly trained BioGottBERT model outperformed GottBERT on both clinical NER tasks. However, training new biomedical models from scratch proved ineffective. The domain-adaptation strategy's potential is currently limited due to a lack of pre-training data. Since general-purpose language models are only marginally inferior to domain-specific models, both options are suitable for developing German-language biomedical applications. General-purpose language models perform remarkably well on biomedical and clinical NLP tasks. If larger corpora become available in the future, domain-adapting these models may improve performances.

Estimating distribution shifts for predicting cross-subject generalization in electroencephalography-based mental workload assessment.

Assessment of mental workload in real-world conditions is key to ensuring the performance of workers executing tasks that demand sustained attention. Previous literature has employed electroencephalography (EEG) to this end despite having observed that EEG correlates of mental workload vary across subjects and physical strain, thus making it difficult to devise models capable of simultaneously presenting reliable performance across users. Domain adaptation consists of a set of strategies that aim at allowing for improving machine learning systems performance on unseen data at training time. Such methods, however, might rely on assumptions over the considered data distributions, which typically do not hold for applications of EEG data. Motivated by this observation, in this work we propose a strategy to estimate two types of discrepancies between multiple data distributions, namely marginal and conditional shifts, observed on data collected from different subjects. Besides shedding light on the assumptions that hold for a particular dataset, the estimates of statistical shifts obtained with the proposed approach can be used for investigating other aspects of a machine learning pipeline, such as quantitatively assessing the effectiveness of domain adaptation strategies. In particular, we consider EEG data collected from individuals performing mental tasks while running on a treadmill and pedaling on a stationary bike and explore the effects of different normalization strategies commonly used to mitigate cross-subject variability. We show the effects that different normalization schemes have on statistical shifts and their relationship with the accuracy of mental workload prediction as assessed on unseen participants at training time.

Weakly Supervised Domain Adaptation for Aspect Extraction via Multilevel Interaction Transfer.

Fine-grained aspect term extraction is an essential subtask in aspect-based opinion analysis. It aims to identify the aspect terms (also known as opinion targets) of a product or service in each sentence. To learn a good aspect extraction model, an expensive annotation process is usually involved to acquire sufficient token-level labels for each domain, which is not realistic. To address this limitation, some previous works propose domain adaptation strategies to transfer knowledge from a sufficiently labeled source domain to unlabeled target domains. However, due to both the difficulty of fine-grained prediction problems and the large domain gap between different domains, the performance is still far from satisfactory. In this work, we conduct a pioneer study on leveraging sentence-level aspect category labels that can be usually available in commercial services, such as review sites or social media to promote token-level transfer for extraction purpose. Specifically, the aspect category information can be used to construct pivot knowledge for transfer with the assumption that the interactions between the sentence-level aspect category and the token-level aspect terms are invariant across domains. To this end, we propose a novel multilevel reconstruction mechanism that aligns both the fine- and coarse-grained information in multiple levels of abstractions. Comprehensive experiments over several benchmark data sets clearly demonstrate that our approach can fully utilize the sentence-level aspect category labels to improve cross-domain aspect term extraction with a large performance gain.

High-temperature augmented neighborhood metric learning for cross-domain fault diagnosis with imbalanced data

The class imbalance in the machine monitoring data and the data distribution discrepancy due to the different working conditions make it difficult to build a strong mechanical fault diagnosis model in practice. In this paper, we proposed a high-temperature augmented neighborhood metric-learning network (HNMN) for cross-domain fault diagnosis with imbalanced data. In the proposed network, fault types are classified by comparing the distance from the sample to the prototype. And neighborhood component analysis-based domain adaptation scheme was presented to reduce the distance of similar samples in source and target domains thus eliminating the domain shift and solving the variable working condition problem. A classification boundary optimization strategy based on high-temperature mechanism was applied to avoid the overlap of feature distributions of minor classes and major classes, and finally solve the class imbalance problem. The performance and superiority of HNMN are evaluated and analyzed using three fault diagnosis cases. The experimental results demonstrate that the proposed network can achieve higher fault diagnosis accuracy compared to the state-of-the-art methods in the class imbalance and variable working condition diagnosis scenarios.

Exposing unseen GAN-generated image using unsupervised domain adaptation

Benefiting from the tremendous success of Generative Adversarial Networks (GANs), we are entering a world where it will be increasingly difficult for people to distinguish whether an image is synthetic, which puts new demands on digital image forensics. To resolve this issue, several fake image detectors based on supervised binary classification have been designed. However, current methods remain vulnerable when testing samples are generated by an unseen GAN model. In this work, an unsupervised domain adaptation strategy is introduced to achieve the detection of GAN-generated images in an unknown domain. A Self-Attention block and novel loss function have been constructed to optimize the domain adaptation process, thus obtaining a better generalization. Experimental results demonstrate that the proposed scheme obtains high detection accuracy on the generalization problem and is robust to common post-processing in reality, which shows that unsupervised methods can enhance the generalization ability of detection models to some extent.

Deep Hash with Optimal Transport-Based Domain Adaptation for Multisite MRI Retrieval

The Internet of Things has a wide range of applications in the medical field. Due to the heterogeneity of medical data generated by different hospitals, it is very important to analyze and integrate data from different institutions. Functional magnetic resonance imaging (fMRI) is widely used in clinical medicine and cognitive neuroscience, while resting-state fMRI (rs-fMRI) can help reveal functional biomarkers of neurological disorders for computer-assisted clinical diagnosis and prognosis. Recently, how to retrieve similar images or case histories from large-scale medical image repositories acquired from multiple sites has attracted widespread attention in the field of intelligent diagnosis of diseases. Although using multisite data effectively helps increase the sample size, it also inevitably introduces the problem of data heterogeneity across sites. To address this problem, we propose a multisite fMRI retrieval (MSFR) method that uses a deep hashing approach and an optimal transport-based domain adaptation strategy to mitigate multisite data heterogeneity for accurate fMRI search. Specifically, for a given target domain site and multiple source domain sites, our approach uses a deep neural network to map the source and target domain data into the latent feature space and minimize their Wasserstein distance to reduce their distribution differences. We then use the source domain data to learn high-quality hash code through a global similarity metric, thereby improving the performance of cross-site fMRI retrieval. We evaluated our method on the publicly available Autism Brain Imaging Data Exchange (ABIDE) dataset. Experimental results show the effectiveness of our method in resting-state fMRI retrieval.

Unsupervised Domain Adaptive Corner Detection in Vehicle Plate Images.

Rectification of vehicle plate images helps to improve the accuracy of license-plate recognition (LPR). It is a perspective-transformation process to project images as if taken from the front geometrically. To obtain the projection matrix, we require the (x, y) coordinates of four corner positions of plates in images. In this paper, we consider the problem of unsupervised domain adaptation for corner detection in plate images. We trained a model with plate images of one country, the source domain, and applied a domain adaptation scheme so that the model is able to work well on the plates of a different country, the target domain. For this study, we created a dataset of 22,096 Korea plate images with corner labels, which are source domain, and 6762 Philippines, which are target domain. To address this problem, we propose a heatmap-based corner-detection model, which outperforms existing scalar-regression methods, and an image classifier for mixed image of source and target images for domain adaptation. The proposed approach achieves better accuracy, which is 19.1% improvement if compared with baseline discriminator-based domain adaptation scheme.

Cross-Language Speech Emotion Recognition Using Bag-of-Word Representations, Domain Adaptation, and Data Augmentation.

To date, several methods have been explored for the challenging task of cross-language speech emotion recognition, including the bag-of-words (BoW) methodology for feature processing, domain adaptation for feature distribution “normalization”, and data augmentation to make machine learning algorithms more robust across testing conditions. Their combined use, however, has yet to be explored. In this paper, we aim to fill this gap and compare the benefits achieved by combining different domain adaptation strategies with the BoW method, as well as with data augmentation. Moreover, while domain adaptation strategies, such as the correlation alignment (CORAL) method, require knowledge of the test data language, we propose a variant that we term N-CORAL, in which test languages (in our case, Chinese) are mapped to a common distribution in an unsupervised manner. Experiments with German, French, and Hungarian language datasets were performed, and the proposed N-CORAL method, combined with BoW and data augmentation, was shown to achieve the best arousal and valence prediction accuracy, highlighting the usefulness of the proposed method for “in the wild” speech emotion recognition. In fact, N-CORAL combined with BoW was shown to provide robustness across languages, whereas data augmentation provided additional robustness against cross-corpus nuance factors.

A transferable neural network method for remaining useful life prediction

Prognostics based on the deep learning model often assume that their training and testing data come from the same equipment with similar working conditions. However, a machine often has specific operating conditions for different tasks, which will cause significant divergence in its measurements. Generally, planned maintenance or model-based fault detections can be done first to collect very few suspension histories when the machine works in a new environment. Few suspension histories can help the prognostic model generalize to new environments. This paper proposes a transductive method to use limited suspension histories in transfer prognostics. Different from reported cross-domain prognostics that only align two-domain histories in a holistic manner, the proposed domain adaptation strategy simultaneously minimizes the distance between both marginal and conditional probability distributions in different domains. If the measurements have a clear degradation manifold, iterative learning will allow the model to get better and better pseudo predictions, thus guiding the prognostic model to learn generalized domain invariant features to deal with different working conditions. A heuristic method and a parallel framework are proposed to verify model parameters and uncertainties. The prognostic performance of the proposed approach is validated by using two case studies.

DADER

Entity resolution (ER) is a core data integration problem that identifies pairs of data instances referring to the same real-world entities, and the state-of-the-art results of ER are achieved by deep learning (DL) based approaches. However, DL-based approaches typically require a large amount of labeled training data (i.e. , matching and non-matching pairs), which incurs substantial manual labeling efforts. In this paper, we introduce DADER , a hands-off deep ER system through domain adaptation. DADER utilizes multiple well-labeled source ER datasets to train a DL-based ER model for a new target ER dataset that does not have any labels or with only a few labels. To address the key challenge of domain shift, DADER judiciously selects labeled entity pairs from the source and then aligns distributions of the source and the target by using six popular domain adaptation strategies. DADER can also harness the users to gather a few labels for further improvement. We have built DADER as an open-sourced Python Library with intuitive APIs and demonstrated its utility on supporting hands-off ER in real-world scenarios.

Convolutional kernel aggregated domain adaptation for intelligent fault diagnosis with label noise

Unsupervised domain adaptation for intelligent fault diagnosis requires a well-annotated source domain to transfer knowledge to an unlabeled target domain, but the ubiquitous source label noise in realistic scenarios remains largely neglected. Recent efforts following adversarial domain adaptation attempt to learn with label noise conditioned on the classifier predictions. However, an essential flaw in the classifier capacity introduces improper adjustments to the loss function. Moreover, they treat domain-specific and domain-invariant representations as a whole, which threats the effectiveness of learning invariant representations. To address these issues, a Convolutional Kernel Aggregated Domain Adaptation (CKADA) strategy is proposed for fault knowledge transfer. Specifically, a convolutional kernel aggregated layer with domain-mixed attention weights is first designed to harness the diverse learning capacities of multiple kernels. Then, by extending such a layer to the classifier, a classification bridge layer is presented to ensure reliable predictions, based on which the side effects of label noise are further relieved through selecting and reusing source samples. Meanwhile, an additional discrimination bridge layer is constructed, which collaborates with the classification bridge layer to assist adversarial domain adaptation. Extensive experiments on three rolling bearing datasets with various types of noisy transfer tasks demonstrate the effectiveness and robustness of CKADA.

Remaining useful life prediction based on transfer multi-stage shrinkage attention temporal convolutional network under variable working conditions

Many data-driven remaining useful life (RUL) prediction methods usually assume that the training and test data are independent and identically distributed. However, the different degradation trends of machines under variable working conditions can lead to problems with disparate distribution of degradation features and difficulties in obtaining the corresponding labels. To address the above problems, this paper proposed a RUL prediction method based on a transfer multi-stage shrinkage attention temporal convolutional network under variable working conditions. Firstly, a shrinkage attention module is designed by using the attention mechanism and shrinkage operation to eliminate the interference of irrelevant information and increase the focus on critical features. Secondly, a multi-stage shrinkage attention temporal convolution block based on a hybrid attention subnetwork and soft thresholding subnetwork is designed to efficiently learn the manifold structure of the input data to capture the degenerate information-rich deep features. Finally, an unsupervised domain adaptation strategy based on representation subspace distance and bases mismatch penalization is proposed to enhance the learning of cross-domain invariant features. The proposed method is experimentally studied on XJTU-SY and FEMTO datasets. The experimental results demonstrate that the effectiveness and accuracy of the proposed method in RUL prediction are higher than other methods.

Deep learning based domain adaptation for mitochondria segmentation on EM volumes

Background and Objective: Accurate segmentation of electron microscopy (EM) volumes of the brain is essential to characterize neuronal structures at a cell or organelle level. While supervised deep learning methods have led to major breakthroughs in that direction during the past years, they usually require large amounts of annotated data to be trained, and perform poorly on other data acquired under similar experimental and imaging conditions. This is a problem known as domain adaptation, since models that learned from a sample distribution (or source domain) struggle to maintain their performance on samples extracted from a different distribution or target domain. In this work, we address the complex case of deep learning based domain adaptation for mitochondria segmentation across EM datasets from different tissues and species. Methods: We present three unsupervised domain adaptation strategies to improve mitochondria segmentation in the target domain based on (1) state-of-the-art style transfer between images of both domains; (2) self-supervised learning to pre-train a model using unlabeled source and target images, and then fine-tune it only with the source labels; and (3) multi-task neural network architectures trained end-to-end with both labeled and unlabeled images. Additionally, to ensure good generalization in our models, we propose a new training stopping criterion based on morphological priors obtained exclusively in the source domain. The code and its documentation are publicly available at https://github.com/danifranco/EM_domain_adaptation. Results: We carried out all possible cross-dataset experiments using three publicly available EM datasets. We evaluated our proposed strategies and those of others based on the mitochondria semantic labels predicted on the target datasets. Conclusions: The methods introduced here outperform the baseline methods and compare favorably to the state of the art. In the absence of validation labels, monitoring our proposed morphology-based metric is an intuitive and effective way to stop the training process and select in average optimal models.

New mode cold start monitoring in industrial processes: A solution of spatial–temporal feature transfer

In actual industrial processes, the working conditions often change, resulting in frequent mode switching. Thus, there are no sufficient samples in the start-up stage of a new mode to build an effective model for anomaly monitoring. Meanwhile, the undesirable delay in collecting more modeling samples has posed a threat for real-time process monitoring. We propose a spatial–temporal feature transfer method to address the new mode cold start monitoring by designing a transfer linear dynamic system (TLDS). TLDS enables us to establish a satisfying monitoring model without requiring many samples from the target mode. Unlike most transfer learning methods, our method features a new domain adaptation strategy that simultaneously transfers the temporal and spatial correlations between the source and target domains instead of aligning the static correlations between the two domains. Thus, it is especially well-suited for the dynamic process industry. Moreover, we use the Kullback–Leibler (KL) divergence to align the state transition and observation generation distributions in two domains and apply the expectation maximization (EM) algorithm to estimate the parameters and states in the TLDS model. The effectiveness of this method is verified through a numerical example and the Tennessee Eastman (TE) process benchmark.