Domain-invariant Representations Research Articles

Dual Face Alignment Learning Network for NIR-VIS Face Recognition

As the most important topic in Heterogeneous Face Recognition (HFR), Near-InfraRed and VISual (NIR-VIS) face recognition has attracted increasing research attentions owing to its potential application in the field of criminal detective cases and multimedia information retrieval. However, due to its dramatic intra-class variations, including modality, pose, occlusion, blurry, lighting, distance, expression, etc, it is very challenging to retain inherent identity information. To address the above issue, we propose a novel Dual Face Alignment Learning (DFAL) algorithm to explore the potential domain-invariant neutral face representations of the cross-modal images. Our model contains three effective components including Feature-level Face Alignment (FFA), Image-level Face Alignment (IFA) and Cross-domain compact Representation (CdR). Firstly, Teacher-Encoder CNNs (TeEn-CNNs) and Student-Encoder CNNs (StEn-CNNs) are designed to encode features for VIS neutral face images and non-neutral face images, and the FFA is introduced to learn neutral face representations by performing feature-level alignment between non-neutral face and VIS neutral face. Secondly, Student-Decoder CNNs (StDe-CNNs) is developed to decode features to restore face images, and the IFA is designed to reconstruct neutral face image by imposing image-level alignment. Notably, the FFA acts as the primary target to learn VIS neutral face representations for cross-view data, while the IFA plays a role in the icing on the cake, i.e., further disentangling domain and residual information through the synthesis process. Finally, the CdR dispels modality features and distills identity features by mining inter-class information, inter-domain information and inter-semantic relationship. The joint scheme enables the elimination of intra-class variations and the purification of identity information. We carry out comprehensive experiments to illustrate the effectiveness of the DFAL approach on three challenging NIR-VIS databases.

Time Series Domain Adaptation via Sparse Associative Structure Alignment

Domain adaptation on time series data is an important but challenging task. Most of the existing works in this area are based on the learning of the domain-invariant representation of the data with the help of restrictions like MMD. However, such extraction of the domain-invariant representation is a non-trivial task for time series data, due to the complex dependence among the timestamps. In detail, in the fully dependent time series, a small change of the time lags or the offsets may lead to difficulty in the domain invariant extraction. Fortunately, the stability of the causality inspired us to explore the domain invariant structure of the data. To reduce the difficulty in the discovery of causal structure, we relax it to the sparse associative structure and propose a novel sparse associative structure alignment model for domain adaptation. First, we generate the segment set to exclude the obstacle of offsets. Second, the intra-variables and inter-variables sparse attention mechanisms are devised to extract associative structure time-series data with considering time lags. Finally, the associative structure alignment is used to guide the transfer of knowledge from the source domain to the target one. Experimental studies not only verify the good performance of our methods on three real-world datasets but also provide some insightful discoveries on the transferred knowledge.

Self-Paced Collaborative and Adversarial Network for Unsupervised Domain Adaptation.

This paper proposes a new unsupervised domain adaptation approach called Collaborative and Adversarial Network (CAN), which uses the domain-collaborative and domain-adversarial learning strategies for training the neural network. The domain-collaborative learning strategy aims to learn domain specific feature representation to preserve the discriminability for the target domain, while the domain adversarial learning strategy aims to learn domain invariant feature representation to reduce the domain distribution mismatch between the source and target domains. We show that these two learning strategies can be uniformly formulated as domain classifier learning with positive or negative weights on the losses. We then design a collaborative and adversarial training scheme, which automatically learns domain specific representations from lower blocks in CNNs through collaborative learning and domain invariant representations from higher blocks through adversarial learning. Moreover, to further enhance the discriminability in the target domain, we propose Self-Paced CAN (SPCAN), which progressively selects pseudo-labeled target samples for re-training the classifiers. We employ a self-paced learning strategy such that we can select pseudo-labeled target samples in an easy-to-hard fashion. Additionally, we build upon the popular two-stream approach to extend our domain adaptation approach for more challenging video action recognition task, which additionally considers the cooperation between the RGB stream and the optical flow stream. We propose the Two-stream SPCAN (TS-SPCAN) method to select and reweight the pseudo labeled target samples of one stream (RGB/Flow) based on the information from the other stream (Flow/RGB) in a cooperative way. As a result, our TS-SPCAN model is able to exchange the information between the two streams. Comprehensive experiments on different benchmark datasets, Office-31, ImageCLEF-DA and VISDA-2017 for the object recognition task, and UCF101-10 and HMDB51-10 for the video action recognition task, show our newly proposed approaches achieve the state-of-the-art performance, which clearly demonstrates the effectiveness of our proposed approaches for unsupervised domain adaptation.

A new deep domain adaptation method with joint adversarial training for online detection of bearing early fault

For online early fault detection of rolling bearings in non-stop scenarios, one of the main concerns is the model bias caused by the distribution shift between offline and online working conditions. Under such concern, how to improve the feature sensitivity to early faults and the robustness of detection model has become a key challenge of improving the effectiveness of online detection. To solve this problem, a new online early fault detection method is proposed in this paper based on a strategy of deep transfer learning. First, a new robust state assessment method is presented. By introducing priori degradation information in the anomaly detection process of the isolated forest algorithm, this method can accurately assess the normal state and early fault state under noise interference. Second, a new deep domain adaptation algorithm is proposed. The algorithm uses the results of state assessment as output labels, and designs a deep domain adaptation neural network for joint adversarial training at feature level and model level simultaneously. Then a domain-invariant feature representation can be extracted from the data of different working conditions, and an online detection model can then be constructed. Comparative experiments are run on two bearing datasets IEEE PHM Challenge 2012 and XJTU-SY, and the results verifies the effectiveness of the proposed method in false alarm number and detection location.

Deep domain adaptation with ordinal regression for pain assessment using weakly-labeled videos

Estimation of pain intensity from facial expressions captured in videos has an immense potential for health care applications. Given the challenges related to subjective variations of facial expressions, and to operational capture conditions, the accuracy of state-of-the-art deep learning (DL) models for recognizing facial expressions may decline. Domain adaptation (DA) has been widely explored to alleviate the problem of domain shifts that typically occur between video data captured across various source (laboratory) and target (operational) domains. Moreover, given the laborious task of collecting and annotating videos, and the subjective bias due to ambiguity among adjacent intensity levels, weakly-supervised learning (WSL) is gaining attention in such applications. State-of-the-art WSL models are typically formulated as regression problems, and do not leverage the ordinal relationship among pain intensity levels, nor the temporal coherence of multiple consecutive frames. This paper introduces a new DL model for weakly-supervised DA with ordinal regression (WSDA-OR) that can be adapted using target domain videos with coarse labels provided on a periodic basis. The WSDA-OR model enforces ordinal relationships among the intensity levels assigned to target sequences, and associates multiple relevant frames to sequence-level labels (instead of a single frame). In particular, it learns discriminant and domain-invariant feature representations by integrating multiple instance learning with deep adversarial DA, where soft Gaussian labels are used to efficiently represent the weak ordinal sequence-level labels from the target domain. The proposed approach was validated using the RECOLA video dataset as fully-labeled source domain data, and UNBC-McMaster shoulder pain video dataset as weakly-labeled target domain data. We have also validated WSDA-OR on BIOVID and Fatigue (private) datasets for sequence level estimation. Experimental results indicate that our proposed approach can significantly improve performance over the state-of-the-art models, allowing to achieve a greater pain localization accuracy. Code is available on GitHub link: https://github.com/praveena2j/WSDAOR.

Unsupervised Domain Adaptation for Cloud Detection Based on Grouped Features Alignment and Entropy Minimization

Most convolutional neural network (CNN)-based cloud detection methods are built upon the supervised learning framework that requires a large number of pixel-level labels. However, it is expensive and time-consuming to manually annotate pixelwise labels for massive remote sensing images. To reduce the labeling cost, we propose an unsupervised domain adaptation (UDA) approach to generalize the model trained on labeled images of source satellite to unlabeled images of the target satellite. To effectively address the domain shift problem on cross-satellite images, we develop a novel UDA method based on grouped features alignment (GFA) and entropy minimization (EM) to extract domain-invariant representations to improve the cloud detection accuracy of cross-satellite images. The proposed UDA method is evaluated on “Landsat- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$8~\rightarrow $ </tex-math></inline-formula> ZY-3” and “GF- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1\rightarrow $ </tex-math></inline-formula> ZY-3” domain adaptation tasks. Experimental results demonstrate the effectiveness of our method against existing state-of-the-art UDA approaches. The code of this paper has been made available online ( <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/nkszjx/grouped-features-alignment</uri> ).

Cluster adaptation networks for unsupervised domain adaptation

Domain adaptation is an important technology for transferring source domain knowledge to new, unseen target domains. Recently, domain adaptation models are applied to learn domain invariant representations by minimizing distribution distance or adversarial training in the feature space. However, existing adversarial domain adaptation methods fail to preserve the data structure in the feature space. In this paper, we propose a novel domain adaptation method called Cluster adaptation Networks (CAN). CAN decreases the domain shift by aligning the category centers of source representations and the cluster centers of target representations in the feature space, which preserves the class-level structure and facilitates the classification of the target domain. Experiments on Digits, Office-Home and ImageCLEF-DA datasets validate the effectiveness of the structure preservation in our model.

Self-Path: Self-Supervision for Classification of Pathology Images With Limited Annotations.

While high-resolution pathology images lend themselves well to 'data hungry' deep learning algorithms, obtaining exhaustive annotations on these images for learning is a major challenge. In this article, we propose a self-supervised convolutional neural network (CNN) framework to leverage unlabeled data for learning generalizable and domain invariant representations in pathology images. Our proposed framework, termed as Self-Path, employs multi-task learning where the main task is tissue classification and pretext tasks are a variety of self-supervised tasks with labels inherent to the input images. We introduce novel pathology-specific self-supervision tasks that leverage contextual, multi-resolution and semantic features in pathology images for semi-supervised learning and domain adaptation. We investigate the effectiveness of Self-Path on 3 different pathology datasets. Our results show that Self-Path with the pathology-specific pretext tasks achieves state-of-the-art performance for semi-supervised learning when small amounts of labeled data are available. Further, we show that Self-Path improves domain adaptation for histopathology image classification when there is no labeled data available for the target domain. This approach can potentially be employed for other applications in computational pathology, where annotation budget is often limited or large amount of unlabeled image data is available.

A New Deep Dual Temporal Domain Adaptation Method for Online Detection of Bearings Early Fault.

With the quick development of sensor technology in recent years, online detection of early fault without system halt has received much attention in the field of bearing prognostics and health management. While lacking representative samples of the online data, one can try to adapt the previously-learned detection rule to the online detection task instead of training a new rule merely using online data. As one may come across a change of the data distribution between offline and online working conditions, it is challenging to utilize the data from different working conditions to improve detection accuracy and robustness. To solve this problem, a new online detection method of bearing early fault is proposed in this paper based on deep transfer learning. The proposed method contains an offline stage and an online stage. In the offline stage, a new state assessment method is proposed to determine the period of the normal state and the degradation state for whole-life degradation sequences. Moreover, a new deep dual temporal domain adaptation (DTDA) model is proposed. By adopting a dual adaptation strategy on the time convolutional network and domain adversarial neural network, the DTDA model can effectively extract domain-invariant temporal feature representation. In the online stage, each sequentially-arrived data batch is directly fed into the trained DTDA model to recognize whether an early fault occurs. Furthermore, a health indicator of target bearing is also built based on the DTDA features to intuitively evaluate the detection results. Experiments are conducted on the IEEE Prognostics and Health Management (PHM) Challenge 2012 bearing dataset. The results show that, compared with nine state-of-the-art fault detection and diagnosis methods, the proposed method can get an earlier detection location and lower false alarm rate.

Domain adaptation with feature and label adversarial networks

Learning a cross-domain representation from labeled source domains to unlabeled target domains is an important research problem in representation learning. Despite the success of traditional adversarial methods, they proposed to align features from each domain only while neglecting the importance of labels, when fooling a special domain discriminator network. Thus, the discriminator of these approaches merely distinguishes whether the generated features are in-domain or not, which may lead to less class-discriminative features. In this paper, by considering the joint distributions of features and labels in both domains, we present Feature and Label Adversarial Networks (FLAN). As a result, FLAN can generate more discriminative features in both domains. Experimental results on standard unsupervised domain adaptation benchmarks have demonstrated that FLAN can outperform the state-of-art domain invariant representation learning methods.

Learning domain invariant and specific representation for cross-domain person re-identification

Person re-identification (re-ID) aims to match person images under different cameras with disjoint views. Although supervised re-ID has achieved great progress, unsupervised cross-domain re-ID remains a challenging work due to domain bias. In this work, we divide cross-domain re-ID task into two phases: domain-invariant features learning and domain-specific features learning. Our contributions are twofold. (i) To achieve domain-invariant features learning, a novel model called Pedestrian General Similarity (PGS) is proposed, which can eliminate two main factors that cause domain bias: image style and background. Compared with the existing re-ID models, PGS has better generalization ability. (ii) A novel pseudo label assignment method named Mutual Nearest Neighbors Pseudo Labeling (MNNPL) is proposed, which calculates pseudo labels based on the similarity between samples in the target domain, and the resulting pseudo labels are used to guide domain-specific feature learning. Extensive experiments are conducted on several large scale datasets, the results show that our method outperforms most published unsupervised cross-domain methods by a large margin.

Fault Diagnosis of Rolling Bearings of Different Working Conditions Based on Multi-Feature Spatial Domain Adaptation

The running state of rolling bearings is complex in operation, and the data are generally collected under different working conditions. However, when single-source domain adaptation is used to model the heterogeneously distributed data obtained under different working conditions, the domain-invariant representations can hardly be used for representation, which directly affects the fault diagnosis rate. To this end, a method for the fault diagnosis of rolling bearings under different working conditions based on multi-feature spatial domain adaptation is proposed. Firstly, all the data from source and target domains are mapped into a feature space to learn the common representations of all domains. Secondly, the data for each pair of source and target domains are mapped into different feature spaces to get the fault feature representations under various working conditions. And the multi-domain adaptation network is used for the domain-specific distribution alignment to learn multiple domain-invariant representations. Thirdly, these representations are used to train multiple domain-specific classifiers, thus obtaining the recognition result for each domain-invariant representation. Finally, the domain-specific decision boundaries predicted by multiple classifiers are employed to align the classifiers’ output of target samples and thus to reduce the influence from different classifiers. The effectiveness and feasibility of this proposed method have been verified by diagnostic experiments conducted according to the rolling bearing data from Case Western Reserve University and Laboratory, respectively.

Multi-Domain Adversarial Feature Generalization for Person Re-Identification.

With the assistance of sophisticated training methods applied to single labeled datasets, the performance of fully-supervised person re-identification (Person Re-ID) has been improved significantly in recent years. However, these models trained on a single dataset usually suffer from considerable performance degradation when applied to videos of a different camera network. To make Person Re-ID systems more practical and scalable, several cross-dataset domain adaptation methods have been proposed, which achieve high performance without the labeled data from the target domain. However, these approaches still require the unlabeled data of the target domain during the training process, making them impractical. A practical Person Re-ID system pre-trained on other datasets should start running immediately after deployment on a new site without having to wait until sufficient images or videos are collected and the pre-trained model is tuned. To serve this purpose, in this paper, we reformulate person re-identification as a multi-dataset domain generalization problem. We propose a multi-dataset feature generalization network (MMFA-AAE), which is capable of learning a universal domain-invariant feature representation from multiple labeled datasets and generalizing it to 'unseen' camera systems. The network is based on an adversarial auto-encoder to learn a generalized domain-invariant latent feature representation with the Maximum Mean Discrepancy (MMD) measure to align the distributions across multiple domains. Extensive experiments demonstrate the effectiveness of the proposed method. Our MMFA-AAE approach not only outperforms most of the domain generalization Person Re-ID methods, but also surpasses many state-of-the-art supervised methods and unsupervised domain adaptation methods by a large margin.

Dynamic Joint Domain Adaptation Network for Motor Imagery Classification.

Electroencephalogram (EEG) has been widely used in brain computer interface (BCI) due to its convenience and reliability. The EEG-based BCI applications are majorly limited by the time-consuming calibration procedure for discriminative feature representation and classification. Existing EEG classification methods either heavily depend on the handcrafted features or require adequate annotated samples at each session for calibration. To address these issues, we propose a novel dynamic joint domain adaptation network based on adversarial learning strategy to learn domain-invariant feature representation, and thus improve EEG classification performance in the target domain by leveraging useful information from the source session. Specifically, we explore the global discriminator to align the marginal distribution across domains, and the local discriminator to reduce the conditional distribution discrepancy between sub-domains via conditioning on deep representation as well as the predicted labels from the classifier. In addition, we further investigate a dynamic adversarial factor to adaptively estimate the relative importance of alignment between the marginal and conditional distributions. To evaluate the efficacy of our method, extensive experiments are conducted on two public EEG datasets, namely, Datasets IIa and IIb of BCI Competition IV. The experimental results demonstrate that the proposed method achieves superior performance compared with the state-of-the-art methods.

Remaining Useful Life Estimation Under Multiple Operating Conditions via Deep Subdomain Adaptation

Transfer learning (TL) and domain adaptation (DA) have aroused the interest of many researchers and are used in the prognostics and health management (PHM) of bearings. However, traditional DA methods tend to lose fine-grained information when aligning the domains globally. Moreover, most DA methods are not suitable for regression tasks, such as remaining useful life (RUL) prediction. This article proposes a deep subdomain adaptive regression network (DSARN), which can align the relevant subdomains in the source and target domains. This method has good cross-domain generalization ability and thus can be used for RUL prediction of bearings under multiple operating conditions. The proposed method takes the multichannel time-frequency representation (TFR) of bearings as the input and uses DSARN to extract the domain-invariant hidden representation and to establish a mapping to RUL. The effectiveness of the proposed method is verified by case studies on bearings. A comparison with other state-of-the-art deep learning (DL) and TL methods also proves the superiority of the proposed method.

Towards NIR-VIS Masked Face Recognition

Near-infrared to visible (NIR-VIS) face recognition is the most common case in heterogeneous face recognition, which aims to match a pair of face images captured from two different modalities. Existing deep learning based methods have made remarkable progress in NIR-VIS face recognition, while it encounters certain newly-emerged difficulties during the pandemic of COVID-19, since people are supposed to wear facial masks to cut off the spread of the virus. We define this task as NIR-VIS masked face recognition, and find it problematic with the masked face in the NIR probe image. First, the lack of masked face data is a challenging issue for the network training. Second, most of the facial parts (cheeks, mouth, nose etc.) are fully occluded by the mask, which leads to a large amount of loss of information. Third, the domain gap still exists in the remaining facial parts. In such scenario, the existing methods suffer from significant performance degradation caused by the above issues. In this paper, we aim to address the challenge of NIR-VIS masked face recognition from the perspectives of training data and training method. Specifically, we propose a novel heterogeneous training method to maximize the mutual information shared by the face representation of two domains with the help of semi-siamese networks. In addition, a 3D face reconstruction based approach is employed to synthesize masked face from the existing NIR image. Resorting to these practices, our solution provides the domain-invariant face representation which is also robust to the mask occlusion. Extensive experiments on three NIR-VIS face datasets demonstrate the effectiveness and cross-dataset-generalization capacity of our method.

Learning Causal Representations for Robust Domain Adaptation

Domain adaptation solves the learning problem in a target domain by leveraging the knowledge in a relevant source domain. While remarkable advances have been made, almost all existing domain adaptation methods heavily require large amounts of unlabeled target domain data for learning domain invariant representations to achieve good generalizability on the target domain. In fact, in many real-world applications, target domain data may not always be available. In this paper, we study the cases where at the training phase the target domain data is unavailable and only well-labeled source domain data is available, called robust domain adaptation. To tackle this problem, under the assumption that causal relationships between features and the class variable are robust across domains, we propose a novel Causal AutoEncoder (CAE), which integrates deep autoencoder and causal structure learning into a unified model to learn causal representations only using data from a single source domain. Specifically, a deep autoencoder model is adopted to learn low-dimensional representations, and a causal structure learning model is designed to separate the low-dimensional representations into two groups: causal representations and task-irrelevant representations. Using three real-world datasets the extensive experiments have validated the effectiveness of CAE compared to eleven state-of-the-art methods.

Semi-Supervised Dual Relation Learning for Multi-Label Classification.

In a real-world scenario, an object could contain multiple tags instead of a single categorical label. To this end, multi-label learning (MLL) emerged. In MLL, the feature distributions are long-tailed and the complex semantic label relation and the long-tailed training samples are the main challenges. Semi-supervised learning is a potential solution. While, existing methods are mainly designed for single class scenario while ignoring the latent label relations. In addition, they cannot well handle the distribution shift commonly existing across source and target domains. To this end, a Semi-supervised Dual Relation Learning (SDRL) framework for multi-label classification is proposed. SDRL utilizes a few labeled samples as well as large scale unlabeled samples in the training stage. It jointly explores the inter-instance feature-level relation and the intra-instance label-level relation even from the unlabeled samples. In our model, a dual-classifier structure is deployed to obtain domain invariant representations. The prediction results from the classifiers are further compared and the most confident predictions are extracted as pseudo labels. A trainable label relation tensor is designed to explicitly explore the pairwise latent label relations and refine the predicted labels. SDRL is able to effectively and efficiently explore the feature-label relation as well as the label-label relation knowledge without any extra semantic knowledge. We evaluated SDRL in general and zero-shot multi-label classification tasks and we concluded that SDRL is superior to other SOTA baselines. Furthermore, extensive ablation studies have been done which reveal the effectiveness of each component in our framework.

Joint Clustering and Discriminative Feature Alignment for Unsupervised Domain Adaptation.

Unsupervised Domain Adaptation (UDA) aims to learn a classifier for the unlabeled target domain by leveraging knowledge from a labeled source domain with a different but related distribution. Many existing approaches typically learn a domain-invariant representation space by directly matching the marginal distributions of the two domains. However, they ignore exploring the underlying discriminative features of the target data and align the cross-domain discriminative features, which may lead to suboptimal performance. To tackle these two issues simultaneously, this paper presents a Joint Clustering and Discriminative Feature Alignment (JCDFA) approach for UDA, which is capable of naturally unifying the mining of discriminative features and the alignment of class-discriminative features into one single framework. Specifically, in order to mine the intrinsic discriminative information of the unlabeled target data, JCDFA jointly learns a shared encoding representation for two tasks: supervised classification of labeled source data, and discriminative clustering of unlabeled target data, where the classification of the source domain can guide the clustering learning of the target domain to locate the object category. We then conduct the cross-domain discriminative feature alignment by separately optimizing two new metrics: 1) an extended supervised contrastive learning, i.e., semi-supervised contrastive learning 2) an extended Maximum Mean Discrepancy (MMD), i.e., conditional MMD, explicitly minimizing the intra-class dispersion and maximizing the inter-class compactness. When these two procedures, i.e., discriminative features mining and alignment are integrated into one framework, they tend to benefit from each other to enhance the final performance from a cooperative learning perspective. Experiments are conducted on four real-world benchmarks (e.g., Office-31, ImageCLEF-DA, Office-Home and VisDA-C). All the results demonstrate that our JCDFA can obtain remarkable margins over state-of-the-art domain adaptation methods. Comprehensive ablation studies also verify the importance of each key component of our proposed algorithm and the effectiveness of combining two learning strategies into a framework.

Knowledge Exchange Between Domain-Adversarial and Private Networks Improves Open Set Image Classification.

Both target-specific and domain-invariant features can facilitate Open Set Domain Adaptation (OSDA). To exploit these features, we propose a Knowledge Exchange (KnowEx) model which jointly trains two complementary constituent networks: (1) a Domain-Adversarial Network (DAdvNet) learning the domain-invariant representation, through which the supervision in source domain can be exploited to infer the class information of unlabeled target data; (2) a Private Network (PrivNet) exclusive for target domain, which is beneficial for discriminating between instances from known and unknown classes. The two constituent networks exchange training experience in the learning process. Toward this end, we exploit an adversarial perturbation process against DAdvNet to regularize PrivNet. This enhances the complementarity between the two networks. At the same time, we incorporate an adaptation layer into DAdvNet to address the unreliability of the PrivNet's experience. Therefore, DAdvNet and PrivNet are able to mutually reinforce each other during training. We have conducted thorough experiments on multiple standard benchmarks to verify the effectiveness and superiority of KnowEx in OSDA.