Domain Gap Research Articles

PAR2Net: End-to-end Panoramic Image Reflection Removal.

In this paper, we investigate the problem of panoramic image reflection removal to relieve the content ambiguity between the reflection layer and the transmission scene. Although a partial view of the reflection scene is attainable in the panoramic image and provides additional information for reflection removal, it is not trivial to directly apply this for getting rid of undesired reflections due to its misalignment with the reflection-contaminated image. We propose an end-to-end framework to tackle this problem. By resolving misalignment issues with adaptive modules, high-fidelity recovery of the reflection layer and the transmission scenes are accomplished. We further propose a new data generation approach that considers the physics-based formation model of mixture images and the in-camera dynamic range clipping to diminish the domain gap between synthetic and real data. Experimental results demonstrate the effectiveness of the proposed method and its applicability for mobile devices and industrial applications.

Automatic Loss Function Search for Adversarial Unsupervised Domain Adaptation

Unsupervised domain adaption (UDA) aims to reduce the domain gap between labeled source and unlabeled target domains. Many prior works exploit adversarial learning that leverages pre-designed discriminators to drive the network for aligning distributions between domains. However, most of them do not consider the degeneration of the domain discriminators caused by the gradually dominating gradients of aligned target samples during training, and they still suffer from the cross-domain semantic mismatch problem in the learned feature space. Hence, this paper attempts to understand and solve both issues from the lens of optimization loss and propose an automatic loss function search for adversarial domain adaptation (ALSDA). First, we extend the common adversarial loss by adding an adjustable hyper-parameter that can re-weight the gradients assigned to target samples, so that the domain discriminator can impose consecutive and influential driving forces for domain alignment. Meanwhile, we upgrade the traditional orthogonality loss with class-wisely adjustable hyper-parameters that can strengthen the cross-domain feature separation. Since manually determining the optimal loss functions requires expensive expert efforts, we leverage the popular AutoML to automatically search for the optimal loss functions from a pre-defined novel and unique search space for UDA. Further, to enable the loss function search when the target domain is unlabeled, we introduce a simple-but-effective entropy-guided search strategy with the aid of REINFORCE learning. Extensive experiments on various typical baselines and benchmark datasets such as Office-Home, Office-31, and Birds-31 have been conducted, and the results validate the generalization and superiority of the proposed ALSDA.

Anatomy-guided domain adaptation for 3D in-bed human pose estimation.

3D human pose estimation is a key component of clinical monitoring systems. The clinical applicability of deep pose estimation models, however, is limited by their poor generalization under domain shifts along with their need for sufficient labeled training data. As a remedy, we present a novel domain adaptation method, adapting a model from a labeled source to a shifted unlabeled target domain. Our method comprises two complementary adaptation strategies based on prior knowledge about human anatomy. First, we guide the learning process in the target domain by constraining predictions to the space of anatomically plausible poses. To this end, we embed the prior knowledge into an anatomical loss function that penalizes asymmetric limb lengths, implausible bone lengths, and implausible joint angles. Second, we propose to filter pseudo labels for self-training according to their anatomical plausibility and incorporate the concept into the Mean Teacher paradigm. We unify both strategies in a point cloud-based framework applicable to unsupervised and source-free domain adaptation. Evaluation is performed for in-bed pose estimation under two adaptation scenarios, using the public SLP dataset and a newly created dataset. Our method consistently outperforms various state-of-the-art domain adaptation methods, surpasses the baseline model by 31%/66%, and reduces the domain gap by 65%/82%. Source code is available at https://github.com/multimodallearning/da-3dhpe-anatomy.

High-Quality Object Detection Method for UAV Images Based on Improved DINO and Masked Image Modeling

The extensive application of unmanned aerial vehicle (UAV) technology has increased academic interest in object detection algorithms for UAV images. Nevertheless, these algorithms present issues such as low accuracy, inadequate stability, and insufficient pre-training model utilization. Therefore, a high-quality object detection method based on a performance-improved object detection baseline and pretraining algorithm is proposed. To fully extract global and local feature information, a hybrid backbone based on the combination of convolutional neural network (CNN) and vision transformer (ViT) is constructed using an excellent object detection method as the baseline network for feature extraction. This backbone is then combined with a more stable and generalizable optimizer to obtain high-quality object detection results. Because the domain gap between natural and UAV aerial photography scenes hinders the application of mainstream pre-training models to downstream UAV image object detection tasks, this study applies the masked image modeling (MIM) method to aerospace remote sensing datasets with a lower volume than mainstream natural scene datasets to produce a pre-training model for the proposed method and further improve UAV image object detection accuracy. Experimental results for two UAV imagery datasets show that the proposed method achieves better object detection performance compared to state-of-the-art (SOTA) methods with fewer pre-training datasets and parameters.

EMOS: Enhanced moving object detection and classification via sensor fusion and noise filtering

AbstractDynamic object detection is essential for ensuring safe and reliable autonomous driving. Recently, light detection and ranging (LiDAR)‐based object detection has been introduced and shown excellent performance on various benchmarks. Although LiDAR sensors have excellent accuracy in estimating distance, they lack texture or color information and have a lower resolution than conventional cameras. In addition, performance degradation occurs when a LiDAR‐based object detection model is applied to different driving environments or when sensors from different LiDAR manufacturers are utilized owing to the domain gap phenomenon. To address these issues, a sensor‐fusion‐based object detection and classification method is proposed. The proposed method operates in real time, making it suitable for integration into autonomous vehicles. It performs well on our custom dataset and on publicly available datasets, demonstrating its effectiveness in real‐world road environments. In addition, we will make available a novel three‐dimensional moving object detection dataset called ETRI 3D MOD.

Structure-preserving feature alignment for old photo colorization

Deep learning techniques have made significant advancements in reference-based colorization by training on large-scale datasets. However, directly applying these methods to the task of colorizing old photos is challenging due to the lack of ground truth and the notorious domain gap between natural gray images and old photos. To address this issue, we propose a novel CNN-based algorithm called SFAC, i.e., Structure-preserving Feature Alignment Colorizer. SFAC is trained on only two images for old photo colorization, eliminating the reliance on big data and allowing direct processing of the old photo itself to overcome the domain gap problem. Our primary objective is to establish semantic correspondence between the two images, ensuring that semantically related objects have similar colors. We achieve this through a feature distribution alignment loss that remains robust to different metric choices. However, utilizing robust semantic correspondence to transfer color from the reference to the old photo can result in inevitable structure distortions. To mitigate this, we introduce a structure-preserving mechanism that incorporates a perceptual constraint at the feature level and a frozen-updated pyramid at the pixel level. Extensive experiments demonstrate the effectiveness of our method for old photo colorization, as confirmed by qualitative and quantitative metrics.

An Improved CycleGAN for Data Augmentation in Person Re-Identification

Person re-identification (ReID) has attracted more and more attention, which is to retrieve interested persons across multiple non-overlapping cameras. Matching the same person between different camera styles has always been an enormous challenge. In the existing work, cross-camera styles images generated by the cycle-consistent generative adversarial network (CycleGAN) only transfer the camera resolution and ambient lighting. The generated images produce considerable redundancy and inappropriate pictures at the same time. Although the data is added to prevent over-fitting, it also makes significant noise, so the accuracy is not significantly improved. In this paper, an improved CycleGAN is proposed to generate images for achieving improved data augmentation. The transfer of pedestrian posture is added at the same time as transferring the image style. It not only increases the diversity of pedestrian posture but also reduces the domain gap caused by the style change between cameras. Besides, through the multi-pseudo regularized label (MpRL), the generated images are assigned virtual tags dynamically in training. Through many experimental evaluations, we have achieved a very high identification accuracy on Market-1501, DukeMTMC-reID, and CUHK03-NP datasets. On the three datasets, the quantitative results of mAP are 96.20%, 93.72%, and 86.65%, and the quantitative results of rank-1 are 98.27%, 95.37%, and 90.71%, respectively. The experimental results fully show the superiority of our proposed method.

Underwater Depth Estimation via Stereo Adaptation Networks

With the fast development and wide application of stereo depth estimation, adequate high-quality stereo training data with groundtruth depth information plays an important role, but is not easily acquired in underwater environments. Therefore, satisfactory performance of depth estimation is difficult to achieve in underwater environments. In addition, the domain gap also leads to the failure of directly applying existing models of terrestrial scene to underwater scene. Therefore, this paper proposes a novel underwater depth estimation network which can infer depth maps from real underwater stereo images in an adaptation manner. The proposed learning pipeline mainly contains three different adaptation modules, i.e., style adaptation, semantic adaptation and disparity range adaptation, to progressively adapt a terrestrial depth estimation model to the underwater domain. Specifically, due to the lack of underwater training data, we first propose a depth-aware stereo image translation network to synthesize stylized underwater stereo images from terrestrial dataset, thus benefiting the effective training of depth estimation network. Then, considering the weak generalization to the real underwater data when only trained on the above synthetic data, we present a self-ensembling semantic adaptation for depth estimation network to minimize the semantic domain discrepancy between synthetic and real underwater data. Meanwhile, we design a disparity range adaptation module to address the problem of disparity range miss-match between both data, thus obtaining more accurate depth predictions for large-disparity-span underwater images. Experimental results show that by integrating the proposed adaptation modules into the off-the-shelf depth estimation backbones, our method successfully achieves superior performance of underwater depth estimation compared to other state-of-the-art methods.

Industrial UAV-Based Unsupervised Domain Adaptive Crack Recognitions: From Database Towards Real-Site Infrastructural Inspections

The defect diagnosis of modern infrastructures is crucial to public safety. In this work, we propose an unsupervised domain adaptive crack recognition framework. To fulfill the unsupervised domain adaptation (UDA) task of cracks recognition in infrastructural inspections, we propose a robust unsupervised domain adaptive learning strategy termed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Crack-DA</i> to increase the generalization capacity of the model in unseen test circumstances. More specifically, we first propose leveraging the self-supervised depth information to help the learning of semantics. And then using the edge information to suppress non-edge background objects and noises. We propose the data augmentation-based consistency to enhance the performance. More importantly, we proposed to use the disparity in depth to evaluate the domain gap in semantics and explicitly consider the domain gap in network optimization. A database composed of a large number of crack images with detailed pixel-level labels for network training is established. Extensive experiments on UAV-captured highway cracks and real-site UAV inspections of building cracks demonstrate the robustness and effectiveness of our proposed domain adaptive crack recognition approach.

Pixel-Level Domain Adaptation for Real-to-Sim Object Pose Estimation

Transferring robotic grasping skills learned from a simulator to the real world is beneficial in reducing the cost of labeling. However, the models trained on synthetic data are brittle when being applied to real-world data due to the domain gap. In this paper, we propose CCM Pixel-DA, a novel real-to-sim approach to unsupervised domain adaptation for object pose estimations, which outperforms conventional domain adaptation methods in preserving structural information, semantic information, and object pose during the transfer. The pipeline decouples domain adaptation and pose estimation, which allows the CCM Pixel-DA method to be integrated into state-of-the-art object pose estimation networks. The proposed method is further integrated into a pipeline for robot grasping. Experimental results on a real-world robot grasping system validate that the system is capable of grasping real-world objects without object pose annotations in the real-world domain.

Unsupervised Pixel-Level Detection of Rail Surface Defects Using Multistep Domain Adaptation

Detection of rail surface defects using deep learning largely relies on supervision with pixel-level ground truths, where the detection accuracy suffers due to unseen image domains. In order to streamline the laborious labeling process and accurately identify defects on rail surfaces a U-Net-based network that integrates spatial and channel attention (SA) modules, multistep domain adaptation (MSDA), and progressive histogram matching (PHM) is developed in the context of pixel-level defects detection in this research. MSDA with an unlabeled intermediate dataset is used to reduce the domain gap underlying different domains. To further improve the adaptation of the transferred model, the PHM method and the self-training method are employed to enhance detection accuracy. Trained in the source domain with ground truths, the deep learning model is then tested in the target domain with no annotations. Experimental results on the RSDD dataset and the Rail-Joint dataset demonstrate the applicability and effectiveness of the proposed approach. By alleviating the domain discrepancy, the developed model drastically enhances the average precision (AP) score of unsupervised detection from 0.103 to 0.861, which shows competitive performance compared with the AP score of 0.895 by the state-of-the-art supervised models. The novelty of this research lies in the development of a hybrid deep learning approach that is able to achieve competitive detection accuracy with a low overlap ratio between domains and few data labeling.

ACNet: Approaching-and-Centralizing Network for Zero-Shot Sketch-Based Image Retrieval

The huge domain gap between sketches and photos and the highly abstract sketch representations pose challenges for sketch-based image retrieval (\underline{SBIR}). The zero-shot sketch-based image retrieval (\underline{ZS-SBIR}) is more generic and practical but poses an even greater challenge because of the additional knowledge gap between the seen and unseen categories. To simultaneously mitigate both gaps, we propose an \textbf{A}pproaching-and-\textbf{C}entralizing \textbf{Net}work (termed "\textbf{ACNet}") to jointly optimize sketch-to-photo synthesis and the image retrieval. The retrieval module guides the synthesis module to generate large amounts of diverse photo-like images which gradually approach the photo domain, and thus better serve the retrieval module than ever to learn domain-agnostic representations and category-agnostic common knowledge for generalizing to unseen categories. These diverse images generated with retrieval guidance can effectively alleviate the overfitting problem troubling concrete category-specific training samples with high gradients. We also discover the use of proxy-based NormSoftmax loss is effective in the zero-shot setting because its centralizing effect can stabilize our joint training and promote the generalization ability to unseen categories. Our approach is simple yet effective, which achieves state-of-the-art performance on two widely used ZS-SBIR datasets and surpasses previous methods by a large margin.

Effects of Motion-Relevant Knowledge From Unlabeled Video to Human-Object Interaction Detection.

The existing works on human-object interaction (HOI) detection usually rely on expensive large-scale labeled image datasets. However, in real scenes, labeled data may be insufficient, and some rare HOI categories have few samples. This poses great challenges for deep-learning-based HOI detection models. Existing works tackle it by introducing compositional learning or word embedding but still need large-scale labeled data or extremely rely on the well-learned knowledge. In contrast, the freely available unlabeled videos contain rich motion-relevant information that can help infer rare HOIs. In this article, we creatively propose a multitask learning (MTL) perspective to assist in HOI detection with the aid of motion-relevant knowledge learning on unlabeled videos. Specifically, we design the appearance reconstruction loss (ARL) and sequential motion mining module in a self-supervised manner to learn more generalizable motion representations for promoting the detection of rare HOIs. Moreover, to better transfer motion-related knowledge from unlabeled videos to HOI images, a domain discriminator is introduced to decrease the domain gap between two domains. Extensive experiments on the HICO-DET dataset with rare categories and the V-COCO dataset with minimum supervision demonstrate the effectiveness of motion-aware knowledge implied in unlabeled videos for HOI detection.

Synthetic‐to‐realistic domain adaptation for cold‐start of rail inspection systems

AbstractRail surface defects are potential danger factors for railway systems, and visual inspection of surface defects plays a vital role in rail maintenance. Recently, the methods based on deep learning have been widely used in rail inspection systems, but such systems often face the problem of a lack of defect samples for training deep learning models at start‐up, which is called the cold‐start problem. It is challenging to obtain sufficient defect samples since defects are sparse and even non‐existent for a running railway system. Therefore, a synthetic‐to‐realistic domain adaptation (SRDA) method is proposed for real‐world rail inspection. SRDA adapts synthetic images to look more realistic for reducing the domain gap between synthetic images and realistic rail surface images and obtains translated images that consist of synthetic defect information and realistic rail background. After that, the translated images are used to train a detector for rail inspection of rail surface defects. In order to make the detector more robust to complex backgrounds, SRDA generates images with the same defect‐level semantics but with different texture appearances and makes the detector align these images in the learned feature space. In addition, the synthetic and realistic rail surface defects (SRRSD) dataset containing 20,662 images is built. The experimental results on SRRSD show that SRDA achieves higher detection performance than other established domain adaption methods with 19.0% for and 26.7% for average precision.

CT2X-IRA: CT to x-ray image registration agent using domain-cross multi-scale-stride deep reinforcement learning

Objective. In computer-assisted minimally invasive surgery, the intraoperative x-ray image is enhanced by overlapping it with a preoperative CT volume to improve visualization of vital anatomical structures. Therefore, accurate and robust 3D/2D registration of CT volume and x-ray image is highly desired in clinical practices. However, previous registration methods were prone to initial misalignments and struggled with local minima, leading to issues of low accuracy and vulnerability. Approach. To improve registration performance, we propose a novel CT/x-ray image registration agent (CT2X-IRA) within a task-driven deep reinforcement learning framework, which contains three key strategies: (1) a multi-scale-stride learning mechanism provides multi-scale feature representation and flexible action step size, establishing fast and globally optimal convergence of the registration task. (2) A domain adaptation module reduces the domain gap between the x-ray image and digitally reconstructed radiograph projected from the CT volume, decreasing the sensitivity and uncertainty of the similarity measurement. (3) A weighted reward function facilitates CT2X-IRA in searching for the optimal transformation parameters, improving the estimation accuracy of out-of-plane transformation parameters under large initial misalignments. Main results. We evaluate the proposed CT2X-IRA on both the public and private clinical datasets, achieving target registration errors of 2.13 mm and 2.33 mm with the computation time of 1.5 s and 1.1 s, respectively, showing an accurate and fast workflow for CT/x-ray image rigid registration. Significance. The proposed CT2X-IRA obtains the accurate and robust 3D/2D registration of CT and x-ray images, suggesting its potential significance in clinical applications.

Decoupled Edge Guidance Network for Automatic Checkout.

Automatic checkout (ACO) aims at correctly generating complete shopping lists from checkout images. However, the domain gap between the single product in training data and multiple products in checkout images endows ACO tasks with a major difficulty. Despite remarkable advancements in recent years, resolving the significant domain gap remains challenging. It is possibly because networks trained solely on synthesized images may struggle to generalize well to realistic checkout scenarios. To this end, we propose a decoupled edge guidance network (DEGNet), which integrates synthesized and checkout images via a supervised domain adaptation approach and further learns common domain representations using a domain adapter. Specifically, an edge embedding module is designed for generating edge embedding images to introduce edge information. On this basis, we develop a decoupled feature extractor that takes original images and edge embedding images as input to jointly utilize image information and edge information. Furthermore, a novel proposal divide-and-conquer strategy (PDS) is proposed for the purpose of augmenting high-quality samples. Through experimental evaluation, DEGNet achieves state-of-the-art performance on the retail product checkout (RPC) dataset, with checkout accuracy (cAcc) results of 93.47% and 95.25% in the average mode of faster RCNN and cascade RCNN frameworks, respectively. Codes are available at https://github.com/yourbikun/DEGNet.

DeepPortraitDrawing: Generating human body images from freehand sketches

Various methods for generating realistic images of objects and human faces from freehand sketches have been explored. However, generating realistic human body images from sketches is still a challenging problem. It is, first because of the sensitivity to human shapes, second because of the complexity of human images caused by body shape and pose changes, and third because of the domain gap between realistic images and freehand sketches. In this work, we present DeepPortraitDrawing, a deep generative framework for converting roughly drawn sketches to realistic human body images. To encode complicated body shapes under various poses, we take a local-to-global approach. Locally, we employ semantic part auto-encoders to construct part-level shape spaces, which are useful for refining the geometry of an input pre-segmented hand-drawn sketch. Globally, we employ a cascaded spatial transformer network to refine the structure of body parts by adjusting their spatial locations and relative proportions. Finally, we use a style-based generator as the global synthesis network for the sketch-to-image translation task which is modulated by segmentation maps for semantic preservation. Extensive experiments have shown that given roughly sketched human portraits, our method produces more realistic images than the state-of-the-art sketch-to-image synthesis techniques.

Deep-Learning-Based Metal Artefact Reduction With Unsupervised Domain Adaptation Regularization for Practical CT Images.

CT metal artefact reduction (MAR) methods based on supervised deep learning are often troubled by domain gap between simulated training dataset and real-application dataset, i.e., methods trained on simulation cannot generalize well to practical data. Unsupervised MAR methods can be trained directly on practical data, but they learn MAR with indirect metrics and often perform unsatisfactorily. To tackle the domain gap problem, we propose a novel MAR method called UDAMAR based on unsupervised domain adaptation (UDA). Specifically, we introduce a UDA regularization loss into a typical image-domain supervised MAR method, which mitigates the domain discrepancy between simulated and practical artefacts by feature-space alignment. Our adversarial-based UDA focuses on a low-level feature space where the domain difference of metal artefacts mainly lies. UDAMAR can simultaneously learn MAR from simulated data with known labels and extract critical information from unlabeled practical data. Experiments on both clinical dental and torso datasets show the superiority of UDAMAR by outperforming its supervised backbone and two state-of-the-art unsupervised methods. We carefully analyze UDAMAR by both experiments on simulated metal artefacts and various ablation studies. On simulation, its close performance to the supervised methods and advantages over the unsupervised methods justify its efficacy. Ablation studies on the influence from the weight of UDA regularization loss, UDA feature layers, and the amount of practical data used for training further demonstrate the robustness of UDAMAR. UDAMAR provides a simple and clean design and is easy to implement. These advantages make it a very feasible solution for practical CT MAR.

A novel one-to-multiple unsupervised domain adaptation framework for abdominal organ segmentation.

Abdominal multi-organ segmentation in multi-sequence magnetic resonance images (MRI) is of great significance in many clinical scenarios, e.g., MRI-oriented pre-operative treatment planning. Labeling multiple organs on a single MR sequence is a time-consuming and labor-intensive task, let alone manual labeling on multiple MR sequences. Training a model by one sequence and generalizing it to other domains is one way to reduce the burden of manual annotation, but the existence of domain gap often leads to poor generalization performance of such methods. Image translation-based unsupervised domain adaptation (UDA) is a common way to address this domain gap issue. However, existing methods focus less on keeping anatomical consistency and are limited by one-to-one domain adaptation, leading to low efficiency for adapting a model to multiple target domains. This work proposes a unified framework called OMUDA for one-to-multiple unsupervised domain-adaptive segmentation, where disentanglement between content and style is used to efficiently translate a source domain image into multiple target domains. Moreover, generator refactoring and style constraint are conducted in OMUDA for better maintaining cross-modality structural consistency and reducing domain aliasing. The average Dice Similarity Coefficients (DSCs) of OMUDA for multiple sequences and organs on the in-house test set, the AMOS22 dataset and the CHAOS dataset are 85.51%, 82.66% and 91.38%, respectively, which are slightly lower than those of CycleGAN(85.66% and 83.40%) in the first two data sets and slightly higher than CycleGAN(91.36%) in the last dataset. But compared with CycleGAN, OMUDA reduces floating-point calculations by about 87 percent in the training phase and about 30 percent in the inference stage respectively. The quantitative results in both segmentation performance and training efficiency demonstrate the usability of OMUDA in some practical scenes, such as the initial phase of product development.

Classification Certainty Maximization for Unsupervised Domain Adaptation

The bi-classifier paradigm is a common practice in unsupervised domain adaptation (UDA), where two classifiers are leveraged to guide the model to learn domain invariant features. Previous approaches only focused on the consistency of the outputs between classifiers, but ignored the classification certainty of each classifier. Therefore, existing methods in some cases may mislead the classifiers into the wrong direction of ambiguous outputs and subsequently undermine the discriminability. To challenge this problem, in this paper we propose Classification Certainty Maximization (CCM) which considers both the joint certainty between classifiers and the individual certainty of each classifier via a novel formulation, and derived their optimal weight ratios by theoretical derivation from the perspective of gradient. In addition, we also propose a dynamic centroid update strategy to mitigate the domain gaps at the feature level. Extensive experiments on four widely used UDA datasets show that CCM performs better than the existing state-of-the-art domain adaptation methods. Notably, our dynamic centroid update strategy can be used as a plug-and-play module for existing bi-classifier domain adaptation methods to boost classification accuracy.