Domain Mismatch Research Articles

Towards collaborative appearance and semantic adaptation for medical image segmentation

This paper proposes a new unsupervised domain adaptation framework, named as Collaborative Appearance and Semantic Adaptation (CASA), for addressing the medical domain mismatch problem. Domain adaptation techniques have become one of the hot topics, especially when applying the established deep neural network into new domains in the medical analysis, i.e., semantic segmentation of medical lesions. To achieve unsupervised domain adaptation, our designed CASA framework could preserve synergistic fusion of adaptation knowledge from the perspectives of appearance and semantic. To be specific, we transform the appearance of medical lesions across domains via a Characterization Transfer Module (CTM), which can mitigate the appearance divergence of medical lesions across domains. Meanwhile, a Representation Transfer Module (RTM) is proposed via incorporating with a conditional generative adversarial network, which could transform features of source lesions to target-like feature, and further narrow the domain-wise distribution gap of underlying semantic knowledge. To the end, a challenging application of medical image segmentation is used to extensively validate the effectiveness of our proposed CASA framework. Various experiment results show its superior performance by a significant margin when comparing to the state-of-the-art domain adaptation methods.

Truncated Models for Probabilistic Weighted Retrieval

Existing probabilistic retrieval models do not restrict the domain of the random variables that they deal with. In this article, we show that the upper bound of the normalized term frequency ( tf ) from the relevant documents is much smaller than the upper bound of the normalized tf from the whole collection. As a result, the existing models suffer from two major problems: (i) the domain mismatch causes data modeling error, (ii) since the outliers have very large magnitude and the retrieval models follow tf hypothesis, the combination of these two factors tends to overestimate the relevance score. In an attempt to address these problems, we propose novel weighted probabilistic models based on truncated distributions. We evaluate our models on a set of large document collections. Significant performance improvement over six existing probabilistic models is demonstrated.

A multi-source ensemble domain adaptation method for rotary machine fault diagnosis

Transfer learning has good ability to transfer knowledge for fault diagnosis under different working condition, while domain mismatches or domain shift can still occur during single-source domain transfer fault diagnosis. To alleviate the problem, a multi-source ensemble domain adaptation method is proposed for rotary machinery fault diagnosis. Firstly, multi-source and target domain anchor adapters are constructed based on class-central samples from multi-source domain. Secondly, multi-source ensemble domain adaptation transfer fault diagnosis model considering the mutual difference between multi-source domain is established to obtain multiple classifiers and prediction results. Then the classifiers with good performance are integrated to achieve final diagnosis model and results by ensemble of anchor adapters. Finally, the performance of the proposed method is verified by two experiments. The results show that the proposed method has ability to learn more comprehensive and general domain invariant diagnosis knowledge, significant diagnosis performance and robustness than other transfer learning methods.

The Domain Mismatch Problem in the Broadcast Speaker Attribution Task

The demand of high-quality metadata for the available multimedia content requires the development of new techniques able to correctly identify more and more information, including the speaker information. The task known as speaker attribution aims at identifying all or part of the speakers in the audio under analysis. In this work, we carry out a study of the speaker attribution problem in the broadcast domain. Through our experiments, we illustrate the positive impact of diarization on the final performance. Additionally, we show the influence of the variability present in broadcast data, depicting the broadcast domain as a collection of subdomains with particular characteristics. Taking these two factors into account, we also propose alternative approximations robust against domain mismatch. These approximations include a semisupervised alternative as well as a totally unsupervised new hybrid solution fusing diarization and speaker assignment. Thanks to these two approximations, our performance is boosted around a relative 50%. The analysis has been carried out using the corpus for the Albayzín 2020 challenge, a diarization and speaker attribution evaluation working with broadcast data. These data, provided by Radio Televisión Española (RTVE), the Spanish public Radio and TV Corporation, include multiple shows and genres to analyze the impact of new speech technologies in real-world scenarios.

Speaker recognition based on deep learning: An overview

Speaker recognition is a task of identifying persons from their voices. Recently, deep learning has dramatically revolutionized speaker recognition. However, there is lack of comprehensive reviews on the exciting progress. In this paper, we review several major subtasks of speaker recognition, including speaker verification, identification, diarization, and robust speaker recognition, with a focus on deep-learning-based methods. Because the major advantage of deep learning over conventional methods is its representation ability, which is able to produce highly abstract embedding features from utterances, we first pay close attention to deep-learning-based speaker feature extraction, including the inputs, network structures, temporal pooling strategies, and objective functions respectively, which are the fundamental components of many speaker recognition subtasks. Then, we make an overview of speaker diarization, with an emphasis of recent supervised, end-to-end, and online diarization. Finally, we survey robust speaker recognition from the perspectives of domain adaptation and speech enhancement, which are two major approaches of dealing with domain mismatch and noise problems. Popular and recently released corpora are listed at the end of the paper.

Non-linear Domain Adaptation in Transfer Evolutionary Optimization

The cognitive ability to learn with experience is a hallmark of intelligent systems. The emerging transfer optimization paradigm pursues such human-like problem-solving prowess by leveraging useful information from various source tasks to enhance optimization efficiency on a related target task. The occurrence of harmful negative transfer is a key concern in this setting, paving the way for recent probabilistic model-based transfer evolutionary algorithms that curb this phenomenon. However, in applications where the source and target domains, i.e., the features of their respective search spaces (e.g., dimensionality) and the distribution of good solutions in those spaces, do not match, narrow focus on curbing negative effects can lead to the conservative cancellation of knowledge transfer. Taking this cue, this paper presents a novel perspective on domain adaptation in the context of evolutionary optimization, inducing positive transfers even in scenarios of source-target domain mismatch. Our first contribution is to establish a probabilistic formulation of domain adaptation, by which source and/or target tasks can be mapped to a common solution representation space in which their discrepancy is reduced. Secondly, a domain adaptive transfer evolutionary algorithm is proposed, supporting both offline construction and online data-driven learning of non-linear mapping functions. The performance of the algorithm is experimentally verified, demonstrating superior convergence rate in comparison to state-of-the-art baselines on synthetic benchmarks and a practical case study in multi-location inventory planning. Our results thus shed light on a new research direction for optimization algorithms that improve their efficacy by learning from heterogeneous experiential priors.

Analytical and numerical evaluation of the interaction energy between screw dislocation and inhomogeneous inclusion

Interaction energy between dislocations and inhomogeneities is an important topic in the study of the trapping mechanism and the motion of dislocations. However, analytical solutions for the interaction energy are difficult to derive because of the complex integral over the inclusion domain and material mismatch, while numerical computations involve singularities from dislocations. The present work proposes an effective iterative computational scheme, via the numerical equivalent inclusion method (NEIM) in conjunction with the two-dimensional fast Fourier transform (2D-FFT) algorithm, for handling the elastic field due to a screw dislocation interacting with an arbitrarily shaped inhomogeneous inclusion. The resultant elastic energy and interaction energy may be conveniently obtained by summing the contributions from the elementary solutions. By employing the EIM, the work also presents a closed-form evaluation for the circular inhomogeneity case, in which the effectiveness of the EIM for the non-uniform equivalent eigenstrains is examined. Benchmark examples are provided to validate the present work, and parametric studies are further conducted to demonstrate the effectiveness of the current scheme.

Deep multiple instance learning for foreground speech localization in ambient audio from wearable devices

Over the recent years, machine learning techniques have been employed to produce state-of-the-art results in several audio related tasks. The success of these approaches has been largely due to access to large amounts of open-source datasets and enhancement of computational resources. However, a shortcoming of these methods is that they often fail to generalize well to tasks from real life scenarios, due to domain mismatch. One such task is foreground speech detection from wearable audio devices. Several interfering factors such as dynamically varying environmental conditions, including background speakers, TV, or radio audio, render foreground speech detection to be a challenging task. Moreover, obtaining precise moment-to-moment annotations of audio streams for analysis and model training is also time-consuming and costly. In this work, we use multiple instance learning (MIL) to facilitate development of such models using annotations available at a lower time-resolution (coarsely labeled). We show how MIL can be applied to localize foreground speech in coarsely labeled audio and show both bag-level and instance-level results. We also study different pooling methods and how they can be adapted to densely distributed events as observed in our application. Finally, we show improvements using speech activity detection embeddings as features for foreground detection.

An investigation of domain adaptation in speaker embedding space for speaker recognition

Speaker recognition continues to grow as a research challenge in the field with expanded application in commercial, forensic, educational and general speech technology interfaces. However, challenges remain, especially for naturalistic audio streams including recordings with mismatch between train and test data (i.e., when train or system development data and enrollment/test data or application data are collected from different sources). Mismatch conditions (Hansen and Hasan, 2015) can be divided into two categories, extrinsic (channel, noise, etc.) and intrinsic (duration, language, and speaker traits including stress, emotion, Lombard effect, vocal effort, accent). Here, we investigate speaker recognition for the domain mismatch problem (intrinsic mismatch) especially for those challenges introduced by NIST (National Institute of Standards and Technology) SRE (speaker recognition evaluation) in 2016 and 2018. The challenges introduced in NIST SRE-16 and SRE-18 include language mismatch between train (used for the development of the system) and enrollment/test (used at the application phase). Here, we develop three alternative speaker embedding systems; i-vector, t-vector (an improved triplet loss solution), and x-vector. In addition, a number of unsupervised and supervised (using pseudo labels) methods are also studied for domain mismatch compensation, especially applied at the back-end level. These include adapted PLDA, adapted discriminant analysis, as well as score normalization and calibration methods using unlabeled in-domain data. We propose new variations to discriminant analysis with support vectors (SVDA) as well. These results confirm that SVDA can measurably improve speaker recognition performance for SRE-16 and SRE-18 tasks respectively by +15% and +8% in terms of min-Cprimary; and for EER the gains are +14% and +16% respectively, using i-vector speaker embeddings as the baseline. These advancements offer promising steps toward addressing speaker recognition in naturalistic audio streams.

Global and Local Texture Randomization for Synthetic-to-Real Semantic Segmentation.

Semantic segmentation is a crucial image understanding task, where each pixel of image is categorized into a corresponding label. Since the pixel-wise labeling for ground-truth is tedious and labor intensive, in practical applications, many works exploit the synthetic images to train the model for real-word image semantic segmentation, i.e., Synthetic-to-Real Semantic Segmentation (SRSS). However, Deep Convolutional Neural Networks (CNNs) trained on the source synthetic data may not generalize well to the target real-world data. To address this problem, there has been rapidly growing interest in Domain Adaption technique to mitigate the domain mismatch between the synthetic and real-world images. Besides, Domain Generalization technique is another solution to handle SRSS. In contrast to Domain Adaption, Domain Generalization seeks to address SRSS without accessing any data of the target domain during training. In this work, we propose two simple yet effective texture randomization mechanisms, Global Texture Randomization (GTR) and Local Texture Randomization (LTR), for Domain Generalization based SRSS. GTR is proposed to randomize the texture of source images into diverse unreal texture styles. It aims to alleviate the reliance of the network on texture while promoting the learning of the domain-invariant cues. In addition, we find the texture difference is not always occurred in entire image and may only appear in some local areas. Therefore, we further propose a LTR mechanism to generate diverse local regions for partially stylizing the source images. Finally, we implement a regularization of Consistency between GTR and LTR (CGL) aiming to harmonize the two proposed mechanisms during training. Extensive experiments on five publicly available datasets (i.e., GTA5, SYNTHIA, Cityscapes, BDDS and Mapillary) with various SRSS settings (i.e., GTA5/SYNTHIA to Cityscapes/BDDS/Mapillary) demonstrate that the proposed method is superior to the state-of-the-art methods for domain generalization based SRSS.

Neutral Cross-Entropy Loss Based Unsupervised Domain Adaptation for Semantic Segmentation.

The generalization performance for semantic segmentation remains a major challenge when the data distributions between the source and target domain mismatch. Unsupervised domain adaptation (UDA) approaches are proposed to mitigate the problem above, among which entropy-minimization-based methods have gained more and more attention. However, the methods merely follow the cluster assumption sharpening the prediction distribution, thus have limited performance improvement. Without additional priors, the entropy loss can easily over-sharpen the prediction distribution, which brings noisy information into the learning process. On the other hand, the gradient of the entropy loss is strongly biased toward easy samples, also leading to limited generalization advances. In this paper, we firstly propose a pixel-level consistency regularization method, which introduces the smoothness prior to the UDA problem. Furthermore, we propose the neutral cross-entropy loss based on the consistency regularization, and reveal that its internal neutralization mechanism mitigates the over-sharpness of entropy minimization via the flatness effect of consistency regularization. We also demonstrate that the gradient bias toward easy samples is inherently tackled via the neutral cross-entropy loss. The experiments show that the proposed method has outperformed state-of-the-art methods in two synthetic-to-real experiments, only using the lightweight network.

Visual-Depth Matching Network: Deep RGB-D Domain Adaptation With Unequal Categories.

Existing domain adaptation (DA) methods generally assume that different domains have identical label space, and the training data are only sampled from a single domain. This unrealistic assumption is quite restricted for real-world applications, since it neglects the more practical scenario, where the source domain can contain the categories that are not shared by the target domain, and the training data can be collected from multiple modalities. In this article, we address a more difficult but practical problem, which recognizes RGB images through training on RGB-D data under the label space inequality scenario. There are three challenges in this task: 1) source and target domains are affected by the domain mismatch issue, which results in that the trained models perform imperfectly on the test data; 2) depth images are absent in the target domain (e.g., target images are captured by smartphones), when the source domain contains both the RGB and depth data. It makes the ordinary visual recognition approaches hardly applied to this task; and 3) in the real world, the source and target domains always have different numbers of categories, which would result in a negative transfer bottleneck being more prominent. Toward tackling the above challenges, we formulate a deep model, called visual-depth matching network (VDMN), where two new modules and a matching component can be trained in an end-to-end fashion jointly to identify the common and outlier categories effectively. The significance of VDMN is that it can take advantage of depth information and handle the domain distribution mismatch under label inequality simultaneously. The experimental results reveal that VDMN exceeds the state-of-the-art performance on various DA datasets, especially under the label inequality scenario.

Learning Class-Aligned and Generalized Domain-Invariant Representations for Speech Emotion Recognition

Although recent research on speech emotion recognition has demonstrated that learning domain-invariant features provide an elegant solution to domain mismatch, the features learned by the existing methods lack generalization capabilities to capture latent information from datasets. We propose two novel domain adaptation methods, the generalized domain adversarial neural network (GDANN) and the class-aligned GDANN (CGDANN), to learn generalized domain-invariant representations for emotion recognition. GDANN and CGDANN, which are derived from multitask learning (MTL), consist of three tasks. The main task is to recognize the emotional category to which the input belongs. The remaining two tasks are auxiliary tasks. One is to use a variational autoencoder to model the input distribution, which encourages the model to learn the distribution of latent representations. The other is to learn the common representations of different domains, for which distinguishing via the domain classifier is difficult. The gradient of the domain classifier guides the shared representations of the source and target domains to approximate each other using a gradient reversal layer. To evaluate the effectiveness of the proposed methods, we conduct several experiments with the IEMOCAP and MSP-IMPROV datasets. The results illustrate that good performance is achieved compared with that of state-of-the-art methods. Notably, CGDANN utilizes a small quantity of labeled target domain samples to align the distribution representation and obtains the best performance among the comparison methods. We further visualize the representations learned by the proposed methods and discover that the representations of the source and target domains converge with a low variance.

Adversarial transfer learning for cross-domain visual recognition

In many practical visual recognition scenarios, feature distributions between source domain and the target domain are quite different, which results in the emergence of general cross-domain visual recognition problems. To address the problems of visual domain mismatch, we propose a novel shallow semi-supervised adversarial transfer learning network, which is called Coupled adversarial transfer Domain Adaptation (CatDA), for distribution alignment between two domains. The proposed CatDA approach is inspired by cycleGAN, but leveraging multiple shallow multilayer perceptrons (MLPs) instead of deep networks. Specifically, our CatDA comprises of two symmetric and slim sub-networks, such that the coupled adversarial learning framework is formulated. With such symmetry of two generators, the input data from source/target domain can be fed into the MLP network for target/source domain generation, supervised by two confrontation oriented coupled discriminators. Notably, in order to avoid the critical flaw of high-capacity of the feature extraction function during domain adversarial training, domain specific loss and domain knowledge fidelity loss are proposed in each generator, such that the effectiveness of the proposed transfer network is guaranteed. Additionally, the essential difference from cycleGAN is that our method aims to generate domain-agnostic and aligned features for domain adaptation and transfer learning rather than synthesize realistic images. We show experimentally on a number of benchmark datasets and the proposed approach achieves competitive performance over state-of-the-art domain adaptation and transfer learning approaches.

Intelligent Fault Diagnosis by Fusing Domain Adversarial Training and Maximum Mean Discrepancy via Ensemble Learning

Nowadays, the industrial Internet of Things (IIoT) has been successfully utilized in smart manufacturing. The massive amount of data in IIoT promote the development of deep learning-based health monitoring for industrial equipment. Since monitoring data for mechanical fault diagnosis collected on different working conditions or equipment have domain mismatch, models trained with training data may not work in practical applications. Therefore, it is essential to study fault diagnosis methods with domain adaptation ability. In this article, we propose an intelligent fault diagnosis method based on an improved domain adaptation method. Specifically, two feature extractors concerning feature space distance and domain mismatch are trained using maximum mean discrepancy and domain adversarial training respectively to enhance feature representation. Since separate classifiers are trained for feature extractors, ensemble learning is further utilized to obtain final results. Experimental results indicate that the proposed method is effective and applicable in diagnosing faults with domain mismatch.

Cross‐domain speaker recognition using domain adversarial siamese network with a domain discriminator

With the widespread use of automatic speaker recognition in realistic world, it suffers a lot when there is a domain mismatch, including channel, language, distance etc. Recent research studies have introduced the adversarial-learning mechanism into deep neural networks to reduce the distribution mismatch between different domains. However, they only aligned the domain distributions between the background training and evaluation data. Few focused on the diverse distribution underlying the enrol and test data. In this Letter, the authors propose a domain adversarial siamese (DAS) network trying to eliminate the domain influence on speech representation. Specifically, they feed a network with speech pairs from the same speaker. Then a domain discriminator is introduced to capture the domain consistence or difference between pairs. Final embeddings become domain-invariant and more speaker-discriminative. A cross-channel data set is sort out from NIST speaker recognition evaluation and more experiments are conducted on AISHELL-Wake-Up-1 data set. Results show that DAS performs equally effective with typical domain adversarial methods, improving results at least on energy efficiency rating. Furthermore, it is proved to be more valid for scenarios such as unbalanced data amount and unknown domain, achieving relatively improvements.

EMSContExt: EMS Protocol-Driven Concept Extraction for Cognitive Assistance in Emergency Response

This paper presents a technique for automated curation of a domain-specific knowledge base or lexicon for resource-constrained domains, such as Emergency Medical Services (EMS) and its application to real-time concept extraction and cognitive assistance in emergency response. The EMS responders often verbalize critical information describing the situations at an incident scene, including patients' physical condition and medical history. Automated extraction of EMS protocol-specific concepts from responders' speech data can facilitate cognitive support through the selection and execution of the proper EMS protocols for patient treatment. Although this task is similar to the traditional NLP task of concept extraction, the underlying application domain poses major challenges, including low training resources availability (e.g., no existing EMS ontology, lexicon, or annotated EMS corpus) and domain mismatch. Hence, we develop EMSContExt, a weakly-supervised concept extraction approach for EMS concepts. It utilizes different knowledge bases and a semantic concept model based on a corpus of over 9400 EMS narratives for lexicon expansion. The expanded EMS lexicon is then used to automatically extract critical EMS protocol-specific concepts from real-time EMS speech narratives. Our experimental results show that EMSContExt achieves 0.85 recall and 0.82 F1-score for EMS concept extraction and significantly outperforms MetaMap, a state-of-the-art medical concept extraction tool. We also demonstrate the application of EMSContExt to EMS protocol selection and execution and real-time recommendation of protocol-specific interventions to the EMS responders. Here, EMSContExt outperforms MetaMap with a 6% increase and six times speedup in weighted recall and execution time, respectively.

Variational Domain Adversarial Learning With Mutual Information Maximization for Speaker Verification

Domain mismatch is a common problem in speaker verification (SV) and often causes performance degradation. For the system relying on the Gaussian PLDA backend to suppress the channel variability, the performance would be further limited if there is no Gaussianity constraint on the learned embeddings. This paper proposes an information-maximized variational domain adversarial neural network (InfoVDANN) that incorporates an InfoVAE into domain adversarial training (DAT) to reduce domain mismatch and simultaneously meet the Gaussianity requirement of the PLDA backend. Specifically, DAT is applied to produce speaker discriminative and domain-invariant features, while the InfoVAE performs variational regularization on the embedded features so that they follow a Gaussian distribution. Another benefit of the InfoVAE is that it avoids posterior collapse in VAEs by preserving the mutual information between the embedded features and the training set so that extra speaker information can be retained in the features. Experiments on both SRE16 and SRE18-CMN2 show that the InfoVDANN outperforms the recent VDANN, which suggests that increasing the mutual information between the embedded features and input features enables the InfoVDANN to extract extra speaker information that is otherwise not possible.

Unsupervised adaptation of PLDA models for broadcast diarization

We present a novel model adaptation approach to deal with data variability for speaker diarization in a broadcast environment. Expensive human annotated data can be used to mitigate the domain mismatch by means of supervised model adaptation approaches. By contrast, we propose an unsupervised adaptation method which does not need for in-domain labeled data but only the recording that we are diarizing. We rely on an inner adaptation block which combines Agglomerative Hierarchical Clustering (AHC) and Mean-Shift (MS) clustering techniques with a Fully Bayesian Probabilistic Linear Discriminant Analysis (PLDA) to produce pseudo-speaker labels suitable for model adaptation. We propose multiple adaptation approaches based on this basic block, including unsupervised and semi-supervised. Our proposed solutions, analyzed with the Multi-Genre Broadcast 2015 (MGB) dataset, reported significant improvements (16% relative improvement) with respect to the baseline, also outperforming a supervised adaptation proposal with low resources (9% relative improvement). Furthermore, our proposed unsupervised adaptation is totally compatible with a supervised one. The joint use of both adaptation techniques (supervised and unsupervised) shows a 13% relative improvement with respect to only considering the supervised adaptation.

Embedding Projection for Targeted Cross-lingual Sentiment: Model Comparisons and a Real-World Study

Sentiment analysis benefits from large, hand-annotated resources in order to train and test machine learning models, which are often data hungry. While some languages, e.g., English, have a vast arrayof these resources, most under-resourced languages do not, especially for fine-grained sentiment tasks, such as aspect-level or targeted sentiment analysis. To improve this situation, we propose a cross-lingual approach to sentiment analysis that is applicable to under-resourced languages and takes into account target-level information. This model incorporates sentiment information into bilingual distributional representations, byjointly optimizing them for semantics and sentiment, showing state-of-the-art performance at sentence-level when combined with machine translation. The adaptation to targeted sentiment analysis on multiple domains shows that our model outperforms other projection-based bilingual embedding methods on binary targetedsentiment tasks. Our analysis on ten languages demonstrates that the amount of unlabeled monolingual data has surprisingly little effect on the sentiment results. As expected, the choice of a annotated source language for projection to a target leads to better results for source-target language pairs which are similar. Therefore, our results suggest that more efforts should be spent on the creation of resources for less similar languages tothose which are resource-rich already. Finally, a domain mismatch leads to a decreased performance. This suggests resources in any language should ideally cover varieties of domains.