Domain Adaptation for the Classification of Remote Sensing Data: An Overview of Recent Advances

Due to more abundant data sources, more various objects of interest, and more time-consuming annotations, there is a large amount of out-of-distribution (OOD) data in the remote sensing field, on which the performance of high-accuracy image segmentation models trained under ideal experimental conditions generally degrades dramatically. Domain adaptation (DA) consequently comes into being, which aims to learn the predictor for the label-scarce target domain of interest with the help of the label-sufficient source domain in the presence of the distribution difference, namely, domain shift, between the two domains. However, the off-the-shelf DA methods for image segmentation not only struggle to cope with the more complex domain shift problems in remote sensing imagery but also almost cannot process heterogeneous data directly without information loss. While the current heterogeneous DA methods mostly still rely on some supervision information from the target domain, which is typically inaccessible in the real world. To overcome these drawbacks, we propose the multilevel heterogeneous unsupervised DA (UDA) method, termed MHDA, which unifies the instance-level DA based on cycle consistency, the feature-level DA based on contrastive learning, and the decision-level DA based on task consistency into a framework to more effectively handle the complex domain shift and heterogeneous data. After that, extensive DA experiments are conducted on the International Society for Photogrammetry and Remote Sensing (ISPRS) dataset, the BigCity dataset constructed by ourselves, and the Wuhan University (WHU) dataset, to explore the effect of each module in MHDA, the necessity of heterogeneous DA, and the effectiveness of multilevel DA. And the results demonstrate that MHDA can achieve superior performance on the remote sensing image segmentation task, compared with several state-of-the-art DA methods.

Various remote sensing applications have widely used domain adaptation (DA) methods. Since it does not need to add human interpretation in the target domain, it can be used in cross-region, multi-temporal, and multi-sensor application scenarios. In order to further optimize the design of the loss function and better address the challenges of DA in remote sensing, in this paper, we propose a new universal DA method named C <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> DA for scene recognition of remote sensing images. It has a comprehensive C <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> criterion for recognizing the "unknown" classes by innovatively fusing confidence, consistency, and certainty of samples to make our network training more efficient. We evaluate the performance of our proposed method based on six transfer tasks on three remote sensing datasets. The evaluation results show that our proposed method achieves an average H-score of 58.44%, significantly higher than other SOTA universal DA methods with an average improvement of 2.32~29.43%. Compared to the baseline ResNet-50, it achieves up to 19.92% improvement, demonstrating that the proposed method outperforms in the universal DA scenario. In the future, we also plan to expand the application of this method to more scenarios.

While domain adaptation (DA) methods have made significant strides in remote sensing community, most current works assume that the source domain labels are accurate. However, limited emphasis has been placed on the scenario where source data are mislabeled with noisy annotations, which is more common in real applications and referred to as noisy DA (NDA). This article formulates remote sensing cross-scene classification on NDA scenarios and proposes a novel network called bilateral adaptation network (BAN), which consists of two parts: 1) forward learning (FL), which utilizes a model learning from the noisy source domain and transfers knowledge to target domain; and 2) backward learning (BL), which utilizes a dual model to acquire knowledge from the target domain and transfer it to source domain. We conduct two parts alternately and adopt a symmetrical Kullback-Leibler (KL) loss to align predictions of the model and its dual model in the same domain. This interactive strategy is able to explore bilateral relationships between domains, implicitly reducing label noise in the source domain. In addition, BAN could serve as a universal paradigm to not only improve the existing NDA methods but also enhance recent DA approaches. Comprehensive evaluations on three publicly available RGB-band remote sensing datasets and two hyperspectral datasets validate the superior effectiveness of our proposed BAN. BAN improves the average accuracy by 6.70%–15.70% on RGB datasets and overall accuracy (OA) by 1.36%–3.14% on hyperspectral datasets with flip-20% noise compared to other state-of-the-art DA and NDA approaches. Promising results indicate the potential of our approach in tackling more general and practical problems with noisy source domain.

This chapter presents an extensive review on the techniques proposed in the recent literature for the classification of remote sensing (RS) images. Automatic classification techniques for obtaining land-cover maps from RS images are usually based on supervised learning methods. Accordingly, we focus our attention on supervised techniques for the classification of different types of RS images acquired by new generation satellite sensors. Initially we analyze the critical problems related to different types of RS data and review the classification techniques that can overcome these problems. Then, the most recent methodological developments related to classification techniques in RS are addressed by focusing on semisupervised learning, active learning and domain adaptation approaches. Finally, the most promising research directions in RS data classification are discussed.

Labeling remote sensing data for classification is labor-intensive and time-consuming. Transfer learning (TL), under such context, is attracting increasing attention as it aims to harness information from data set of other regions where labels are readily available. The central topic of concern is to homogenize the large disparities of feature distribution of different data set through domain adaptation (DA). This article proposes a novel DA method for unsupervised TL, namely, multikernel jointly domain matching (MKJDM), which by definition considers multiple kernels as opposed to the currently popular single-kernel methods for measuring the distances between distributions. The single-kernel methods minimize the distances of feature distribution between the source domain (data set with training labels) and the target domain (data set to be classified) through, for example, maximum mean discrepancy (MMD) metric, formed under a kernel function mapping, while the multikernel version (MK-MMD) uses different kernel functions to encapsulate multiple aspects of distribution discrepancies, and is, therefore, more capable of distance minimization. Our MKJDM implementation also considers simultaneously aligning marginal and class conditional distributions and reweight for each instance, which further improves the performance. Two experiments performed on remote sensing images and multi-modal data sets (i.e., Orthophoto and Digital Surface Models), with regions of different countries with distinctly different land patterns serving as source and target domain data, show that the overall accuracies are improved by 37.28% and 46.62% after applications of our MKJDM method. An additional comparative experiment with five state-of-the-art DA methods also demonstrates that our method achieves the best performance.

Traditional remote sensing (RS) image classification methods heavily rely on labeled samples for model training. When labeled samples are unavailable or labeled samples have different distributions from that of the samples to be classified, the classification model may fail. The cross-domain or cross-scene remote sensing image classification is developed for this case where an existing image for training and an unknown image from different scenes or domains for classification. The distribution inconsistency problem may be caused by the differences in acquisition environment conditions, acquisition scene, acquisition time or/and changing sensors. To cope with the cross-domain remote sensing image classification problem, many domain adaptation (DA) techniques have been developed. In this article, we review DA methods in the fields of RS, especially hyperspectral image classification, and provide a survey of DA methods into traditional shallow DA methods (e.g., instance-based, feature-based, and classifier-based adaptations) and recently developed deep DA methods (e.g., discrepancy-based and adversarial-based adaptations).

Recent advances in deep learning have dramatically improved the performance of content-based remote sensing image retrieval (CBRSIR) with the same distribution of training set (source domain) and test set (target domain). In fact, their distributions are inconsistent in most cases, which can lead to a dramatic decrease in retrieval performance. Currently, some unsupervised domain adaptation (DA) methods for other remote sensing applications have been proposed to eliminate the inconsistency. However, the current unsupervised DA methods do not make full use of the target domain’s distribution characteristics when delineating its decision boundary. This tends to degrade the cross-domain retrieval performance. In this article, a pseudo-label consistency learning-based unsupervised DA method (PCLUDA) is proposed for cross-domain CBRSIR. Our PCLUDA method minimizes the difference in probability distribution between the target domain and its perturbed output by a pseudo-label self-training and consistency regularization strategy, followed by adjusting the target domain’s decision boundaries to the low-density region. Besides, minimize class confusion (MCC) is introduced to reduce negative transfer caused by large intraclass variance of RSIs. Two cross-domain datasets with 12 cross-domain scenarios are constructed based on six open access datasets to measure DA methods. Experimental results show that our PCLUDA method achieves superior retrieval performances with average retrieval precision improvement by 4.9%–32.3% compared with eight state-of-the-art DA approaches in complex cross-domain scenarios. Furthermore, other experimental results indicate that our PCLUDA can also reach optimal retrieval performances in different kinds of deep learning networks [i.e., vision transformer (ViT) and convolutional neural networks (CNNs)].

Although enormous Domain Adaptation (DA) approaches have been proposed for cross-scene hyperspectral image (HSI) classification, majority DA methods strongly depend on much prior knowledge of the association among the label sets of source and target domains (encompassing closed set, partial and open set DA), thereby significantly hindering their applications. Realistic application scenarios often require knowledge transfer between domains without restrictions on the label space, which is called Universal Domain Adaptation (UniDA). In this paper, we propose HyUniDA, which is the first attempt to address UniDA scenario from HSIs. HyUniDA contains two major parts: the Shared Semantic Pairing (SSP) and Domain Similarity Score (DSS). We group both source and target domains to form discriminative clusters. The SSP identifies pairs of clusters that have coincident semantic features as the common classes. By examining the consistency level of samples across source and target domains, DSS can estimate the quantity of target clusters and generate distinct clusters without prior knowledge. Meanwhile, we apply the contrastive domain discrepancy to alleviate the offset of samples distribution, with a representative regularizer to assist distinguish target domain clusters. We evaluate our proposed method on three transfer learning tasks for six typical HSI datasets, it turns out that our proposed method yields 3.83%~37.57% improvements compared to other state-of-the-art DA methods.

To train a classifier on a specific domain, often called the target domain, we need labeled data. However, there might be non availability of the labeled data in this domain. In this scenario, we look for a related domain called the source domain, where availability of labeled data is abundant in number. The lack of availability of labeled data in the target domain poses a serious problem and several domain adaptation (DA) approaches have been put forward to cope up with this problem. Existing DA methods seek a subspace common between both the domains (source and target domains) where the distribution difference is minimal and perform manual parameter sensitivity tests to find apposite value of each parameter for their respective objective function. However, for distorted original data, obtaining a common subspace is a challenging task and condensing manual parameter sensitivity testing is also costly and a time-intensive process. To overcome these challenges, some DA methods consider particle swarm optimization (PSO) technique. However, none of the existing DA methods simultaneously tackle these challenges. Therefore, in this paper, we put forward a method called Feature and Parameter Selection approach for visual Domain Adaptation (FPSDA) to address these challenges. In FPSDA, a suitable subset of features across both the domains and an apposite value of each parameter are simultaneously chosen using a PSO approach. Moreover, to guide the PSO, the objective functions of Joint Geometrical and Statistical Alignment (JGSA) [1] method along with preserving original similarity of data is considered as an objective function for our proposed approach. Full Scale experiments on benchmark datasets for cross-domain adaptation verify that FPSDA performs better than many state-of-the-art classic machine learning and domain adaptation approaches.

Effective Prognostics and Health Management (PHM) relies on accurate prediction of the Remaining Useful Life (RUL). Data-driven RUL prediction techniques rely heavily on the representativeness of the available time-to-failure trajectories. Therefore, these methods may not perform well when applied to data from new units of a fleet that follow different operating conditions than those they were trained on. This is also known as domain shifts. Domain adaptation (DA) methods aim to address the domain shift problem by extracting domain invariant features. However, DA methods do not distinguish between the different phases of operation, such as steady states or transient phases. This can result in misalignment due to under- or over-representation of different operation phases. This paper proposes two novel DA approaches for RUL prediction based on an adversarial domain adaptation framework that considers the different phases of the operation profiles separately. The proposed methodologies align the marginal distributions of each phase of the operation profile in the source domain with its counterpart in the target domain. The effectiveness of the proposed methods is evaluated using the New Commercial Modular Aero-Propulsion System (N-CMAPSS) dataset, where sub-fleets of turbofan engines operating in one of the three different flight classes (short, medium, and long) are treated as separate domains. The experimental results show that the proposed methods improve the accuracy of RUL predictions compared to current state-of-the-art DA methods.

In the existing unsupervised domain adaptation (UDA) methods for remote sensing images (RSIs) semantic segmentation, class symmetry is a widely followed ideal assumption, where the source and target RSIs have exactly the same class space. In practice, however, it is often very difficult to find a source RSI with exactly the same classes as the target RSI. More commonly, there are multiple source RSIs available. And there is always an intersection or inclusion relationship between the class spaces of each source–target pair, which can be referred to as class asymmetry. Nevertheless, the class asymmetry domain adaptation segmentation of RSIs with multiple sources has not yet been explored. To this end, a novel class asymmetry RSIs domain adaptation method is proposed for the first time in this article, which consists of four key components. First, a multibranch segmentation network is built to learn an expert for each source RSI. Second, a novel collaborative learning method with the cross-domain mixing strategy is proposed, to supplement the class information for each source while achieving the domain adaptation of each source–target pair. Third, a pseudolabel generation strategy is proposed to effectively combine the strengths of different experts, which can be flexibly applied to two cases where the source class union is equal to or includes the target class set. Fourth, a multiview-enhanced knowledge integration module is developed for high-level knowledge routing and transfer from multiple domains to target predictions. The experimental results of six different class settings on airborne and spaceborne RSIs show that the proposed method can effectively perform the multisource domain adaptation in the case of class asymmetry, and the obtained segmentation performance of target RSIs is significantly better than the existing relevant methods.

Modelling species distributions with remote sensing data: bridging disciplinary perspectives

With the rapid development of the remote sensing monitoring and computer vision technology, the deep learning method has made a great progress to achieve applications such as earth observation, climate change and even space exploration. However, the model trained on existing data cannot be directly used to handle the new remote sensing data, and labeling the new data is also time-consuming and labor-intensive. Unsupervised Domain Adaptation (UDA) is one of the solutions to the aforementioned problems of labeled data defined as the source domain and unlabeled data as the target domain, i.e., its essential purpose is to obtain a well-trained model and tackle the problem of data distribution discrepancy defined as the domain shift between the source and target domain. There are a lot of reviews that have elaborated on UDA methods based on natural data, but few of these studies take into consideration thorough remote sensing applications and contributions. Thus, in this paper, in order to explore the further progress and development of UDA methods in remote sensing, based on the analysis of the causes of domain shift, a comprehensive review is provided with a fine-grained taxonomy of UDA methods applied for remote sensing data, which includes Generative training, Adversarial training, Self-training and Hybrid training methods, to better assist scholars in understanding remote sensing data and further advance the development of methods. Moreover, remote sensing applications are introduced by a thorough dataset analysis. Meanwhile, we sort out definitions and methodology introductions of partial, open-set and multi-domain UDA, which are more pertinent to real-world remote sensing applications. We can draw the conclusion that UDA methods in the field of remote sensing data are carried out later than those applied in natural images, and due to the domain gap caused by appearance differences, most of methods focus on how to use generative training (GT) methods to improve the model’s performance. Finally, we describe the potential deficiencies and further in-depth insights of UDA in the field of remote sensing.

There is a large amount of out-of-distribution data (OOD) in remote sensing, which hinders high-accuracy segmentation models under the assumption of independent identical distribution (i.i.d.) from stable and reliable performance in real-world remote sensing applications. And Domain Adaptation (DA) is presented to seamlessly extend classifiers to the label-scarce target domain in the presence of the label-sufficient source domain with different data distributions. However, given that the domain shift, i.e. the distribution difference between the two domains, is more serious in remote sensing images, the current DA methods for image segmentation in Computer Vision (CV) typically perform unsatisfactorily in remote sensing, even suffering from the negative domain alignment. To this end, this paper proposes the Self-Training Adversarial Domain Adaptation (STADA) method for remote sensing image segmentation, which not only performs adversarial learning to extract domain-invariant features, but also implements Self-Training using pseudo-labels in the target domain denoised by the conditional adversarial loss for classifier adaptation. The ISPRS and WHU datasets are employed to conduct extensive experiments to investigate the effectiveness of STADA and the specific effect of its each DA component. And the experimental results demonstrate that STADA outperforms other state-of-the-art DA methods in the remote sensing image segmentation task.

Cross-database micro-expression recognition (CDMER) is one of recently emerging and interesting problem in micro-expression analysis. CDMER is more challenging than the conventional micro-expression recognition (MER), because the training and testing samples in CDMER come from different micro-expression databases, resulting in inconsistency of the feature distributions between the training and testing sets. In this paper, we contribute to this topic from three aspects. First, we establish a CDMER experimental evaluation protocol aiming to allow the researchers to conveniently work on this topic and evaluate their proposed methods under the same standard. Second, we conduct benchmark experiments by using NINE state-of-the-art domain adaptation (DA) methods and SIX popular spatiotemporal descriptors for investigating CDMER problem from two different perspectives. Third, we propose a novel DA method called region selective transfer regression (RSTR) to deal with the CDMER task. The overall superior performance of RSTR over the state-of-the-art DA methods demonstrates that taking into consideration the facial local region information used in RSTR contributes to developing effective DA methods for dealing with CDMER problem.