Unsupervised Image Translation Research Articles

SEIT: Structural Enhancement for Unsupervised Image Translation in Frequency Domain

For the task of unsupervised image translation, transforming the image style while preserving its original structure remains challenging. In this paper, we propose an unsupervised image translation method with structural enhancement in frequency domain named SEIT. Specifically, a frequency dynamic adaptive (FDA) module is designed for image style transformation that can well transfer the image style while maintaining its overall structure by decoupling the image content and style in frequency domain. Moreover, a wavelet-based structure enhancement (WSE) module is proposed to improve the intermediate translation results by matching the high-frequency information, thus enriching the structural details. Furthermore, a multi-scale network architecture is designed to extract the domain-specific information using image-independent encoders for both the source and target domains. The extensive experimental results well demonstrate the effectiveness of the proposed method.

Unsupervised image translation with distributional semantics awareness

Unsupervised image translation (UIT) studies the mapping between two image domains. Since such mappings are under-constrained, existing research has pursued various desirable properties such as distributional matching or two-way consistency. In this paper, we re-examine UIT from a new perspective: distributional semantics consistency, based on the observation that data variations contain semantics, e.g., shoes varying in colors. Further, the semantics can be multi-dimensional, e.g., shoes also varying in style, functionality, etc. Given two image domains, matching these semantic dimensions during UIT will produce mappings with explicable correspondences, which has not been investigated previously. We propose distributional semantics mapping (DSM), the first UIT method which explicitly matches semantics between two domains. We show that distributional semantics has been rarely considered within and beyond UIT, even though it is a common problem in deep learning. We evaluate DSM on several benchmark datasets, demonstrating its general ability to capture distributional semantics. Extensive comparisons show that DSM not only produces explicable mappings, but also improves image quality in general.

FAEC‐GAN: An unsupervised face‐to‐anime translation based on edge enhancement and coordinate attention

AbstractAnimation is a widely loved artistic form with high abstraction and powerful expression. The task of image translation from face to anime involves complex geometric and texture transformations, and requires the generated images with clear lines. The existing unsupervised image translation frameworks are often ineffective for this task. According to the characteristics of animation image, we propose an animation translation method based on edge enhancement and coordinate attention, which is called FAEC‐GAN. We design a novel edge discrimination network to identify the edge features of images, so that the generated anime images can present clear and coherent lines. And the coordinate attention module is introduced in the encoder to adapt the model to the geometric changes in translation, so as to produce more realistic animation images. In addition, our method combines the focal frequency loss and pixel loss, which can pay attention to both the frequency domain information and pixel information of the generated image to improve the visual effect of the image. The experimental results demonstrate that FAEC‐GAN is superior to the state‐of‐the‐art methods in the task of face‐to‐animation image translation.

Unsupervised Image Translation Using Multi-Scale Residual GAN

Image translation is a classic problem of image processing and computer vision for transforming an image from one domain to another by learning the mapping between an input image and an output image. A novel Multi-scale Residual Generative Adversarial Network (MRGAN) based on unsupervised learning is proposed in this paper for transforming images between different domains using unpaired data. In the model, a dual generater architecture is used to eliminate the dependence on paired training samples and introduce a multi-scale layered residual network in generators for reducing semantic loss of images in the process of encoding. The Wasserstein GAN architecture with gradient penalty (WGAN-GP) is employed in the discriminator to optimize the training process and speed up the network convergence. Comparative experiments on several image translation tasks over style transfers and object migrations show that the proposed MRGAN outperforms strong baseline models by large margins.

Cross-domain object detection using unsupervised image translation

Unsupervised domain adaptation for object detection addresses the adaption of detectors trained in a source domain to work accurately in an unseen target domain. Recently, methods approaching the alignment of the intermediate features proven to be promising, achieving state-of-the-art results. However, these methods are laborious to implement and hard to interpret. Although promising, there is still room for improvements to close the performance gap toward the upper-bound (when training with the target data). In this work, we propose a method to generate an artificial dataset in the target domain to train an object detector. We employed two unsupervised image translators (CycleGAN and an AdaIN-based model) using only annotated data from the source domain and non-annotated data from the target domain. Our key contributions are the proposal of a less complex yet more effective method that also has an improved interpretability. Results on real-world scenarios for autonomous driving show significant improvements, outperforming state-of-the-art methods in most cases, further closing the gap toward the upper-bound.

HQA‐Trans: An end‐to‐end high‐quality‐awareness image translation framework for unsupervised cross‐domain pedestrian detection

Unsupervised cross-domain pedestrian detection has attracted attention in recent years. Although some works adopted unsupervised image translation frameworks to generate an intermediate domain to narrow the gap between source and target domains, the images in the intermediate domain tend to be distorted due to the instability of the generation network. In this work, we propose a new framework to improve the image quality of the generated intermediate domain via an end-to-end translation framework. First, an image quality assessment index is adopted and adjusted appropriately. The part that controls the image quality is kept, and the part that adversely affects the domain style translation is discarded. Secondly, the adjusted image quality assessment index is integrated into the unsupervised image translation framework, where a new loss with the index's weight is proposed. An end-to-end high-quality-awareness image translation framework is constructed to generate a high-quality intermediate domain directly through this process. Finally, the intermediate domain with high-quality images is applied for cross-domain pedestrian detection. Experimental results on benchmark datasets show that the proposed framework can effectively improve unsupervised cross-domain pedestrian detection performance. Compared with some state-of-the-art works, the proposed framework can also achieve superior performance under miss rate metrics.

AFF‐UNIT: Adaptive feature fusion for unsupervised image‐to‐image translation

The task of image-to-image translation is to generate images closer to the target domain style while preserving the significant features of the original image. This paper contends an adaptive feature fusion method for unsupervised image translation. The proposed architecture, termed as AFF-UNIT, is based on a compact network structure to further improve the quality of generated images. First of all, a feature extraction module based on an adaptive feature fusion method is proposed, which combines low-level fine-grained information and high-level semantic information to obtain feature maps with richer information. At the same time, a feature-similarity loss is proposed to guide the feature extraction module to extract features that are more conducive to improving the translation result. In addition, AFF-UNIT reuses the feature extraction module in the generator and discriminator to simplify the framework. Extensive experiments on five popular benchmarks demonstrate the superior performance of AFF-UNIT over state-of-the-art methods in terms of FID, KID, IS, and also human preference. Comprehensive ablation studies are also carried out to isolate the validity of each proposed component.

Multi-modal image synthesis combining content-style adaptive normalization and attentive normalization

We propose a novel unsupervised image translation model following an end-to-end manner,which incorporates Content-Style Adaptive Normalization(CSAN) and Attentive Normalization(AN). First of all, a new attentive normalization is applied for the first time in the style transfer task, which is an improvement and supplement to the traditional instance normalization, it helps to guide the model to pay more attention to the key areas in image translation, while ignoring the secondary areas. Secondly, our proposed CSAN function absorbs not only information of style codes, but also that of content codes. Compared with Adaptive Instance Normalization(AdaIN), CSAN is more favorable to retain content information of input images. In addition, CSAN can help the attention mechanism to flexibly control the amount of change in texture and shape of input images. Finally, a series of comparative experiments and qualitative and quantitative evaluations on the challenging datasets prove that the proposed model is superior and more advanced than State-Of-The-Art(SOTA) in terms of visual quality, diversity,semantic integrity, and style reflection of generated images.

Deep Learning Thermal Image Translation for Night Vision Perception

Context enhancement is critical for the environmental perception in night vision applications, especially for the dark night situation without sufficient illumination. In this article, we propose a thermal image translation method, which can translate thermal/infrared (IR) images into color visible (VI) images, called IR2VI. The IR2VI consists of two cascaded steps: translation from nighttime thermal IR images to gray-scale visible images (GVI), which is called IR-GVI; and the translation from GVI to color visible images (CVI), which is known as GVI-CVI in this article. For the first step, we develop the Texture-Net, a novel unsupervised image translation neural network based on generative adversarial networks. Texture-Net can learn the intrinsic characteristics from the GVI and integrate them into the IR image. In comparison with the state-of-the-art unsupervised image translation methods, the proposed Texture-Net is able to address some common challenges, e.g., incorrect mapping and lack of fine details, with a structure connection module and a region-of-interest focal loss. For the second step, we investigated the state-of-the-art gray-scale image colorization methods and integrate the deep convolutional neural network into the IR2VI framework. The results of the comprehensive evaluation experiments demonstrate the effectiveness of the proposed IR2VI image translation method. This solution will contribute to the environmental perception and understanding in varied night vision applications.

Simplified unsupervised image translation for semantic segmentation adaptation

Image to image translation achieves superior performance with the advent of generative adversarial networks. In this paper, we propose a Simplified Unsupervised Image Translation (SUIT) model for domain adaptation on semantic segmentation. We adopt adversarial training for superior image generation, and design a novel semantic-content loss to enhance visual appearance preservation. Thus, the high-fidelity generated images with target-style can help the model generalize to the target domain. Besides, the semantic-content loss contains two components, which focus on label- and content-consistency, respectively. Both of them can be derived from existing modules of SUIT, which makes it simple yet suitable for domain adaptation on semantic segmentation tasks. Meanwhile, since the transformation network (generator) is decoupled from the segmentation network, the former can be easily transplanted to other semantic segmentation models. Extensive experimental results demonstrate that these translated images within SUIT can significantly improve performance of the model on the target domain, and our model with FCN8s-VGG16 architecture achieves around 13 percentage points improvement in terms of mIoU on multiple semantic segmentation adaptation benchmarks.

Unsupervised image translation using adversarial networks for improved plant disease recognition

Acquisition of data in task-specific applications of machine learning like plant disease recognition is a costly endeavor owing to the requirements of professional human diligence and time constraints. In this paper, we present a pipeline that uses GANs in an unsupervised image translation environment to improve learning with respect to the data distribution in a plant disease dataset, reducing the partiality introduced by acute class imbalance and hence shifting the classification decision boundary towards better performance. The empirical analysis of our method is demonstrated on a limited dataset of 2789 tomato plant disease images, highly corrupted with an imbalance in the 9 disease categories. First, we extend the state of the art for the GAN-based image-to-image translation method by enhancing the perceptual quality of the generated images and preserving the semantics. We introduce AR-GAN, where in addition to the adversarial loss, our synthetic image generator optimizes on Activation Reconstruction loss (ARL) function that optimizes feature activations against the natural image. We present visually more compelling synthetic images in comparison to most prominent existing models and evaluate the performance of our GAN framework in terms of various datasets and metrics. Second, we evaluate the performance of a baseline convolutional neural network classifier for improved recognition using the resulting synthetic samples to augment our training set and compare it with the classical data augmentation scheme. We observe a significant improvement in classification accuracy (+5.2%) using generated synthetic samples as compared to (+0.8%) increase using classic augmentation in an equal class distribution environment.

Show, Attend, and Translate: Unsupervised Image Translation With Self-Regularization and Attention.

Image translation between two domains is a class of problems aiming to learn mapping from an input image in the source domain to an output image in the target domain. It has been applied to numerous applications, such as data augmentation, domain adaptation, and unsupervised training. When paired training data is not accessible, image translation becomes an ill-posed problem. We constrain the problem with the assumption that the translated image needs to be perceptually similar to the original image and also appears to be drawn from the new domain, and propose a simple yet effective image translation model consisting of a single generator trained with a self-regularization term and an adversarial term. We further notice that the existing image translation techniques are agnostic to the subjects of interest and often introduce unwanted changes or artifacts to the input. Thus, we propose to add an attention module to predict an attention map to guide the image translation process. The module learns to attend to key parts of the image while keeping everything else unaltered, essentially avoiding undesired artifacts or changes. Extensive experiments and evaluations show that our model while being simpler, achieves significantly better performance than existing image translation methods.