GAN-based Methods Research Articles

Exploring dual energy CT synthesis in CBCT-based adaptive radiotherapy and proton therapy: application of denoising diffusion probabilistic models

Background.Adaptive radiotherapy (ART) requires precise tissue characterization to optimize treatment plans and enhance the efficacy of radiation delivery while minimizing exposure to organs at risk. Traditional imaging techniques such as cone beam computed tomography (CBCT) used in ART settings often lack the resolution and detail necessary for accurate dosimetry, especially in proton therapy.Purpose.This study aims to enhance ART by introducing an innovative approach that synthesizes dual-energy computed tomography (DECT) images from CBCT scans using a novel 3D conditional denoising diffusion probabilistic model (DDPM) multi-decoder. This method seeks to improve dose calculations in ART planning, enhancing tissue characterization.Methods.We utilized a paired CBCT-DECT dataset from 54 head and neck cancer patients to train and validate our DDPM model. The model employs a multi-decoder Swin-UNET architecture that synthesizes high-resolution DECT images by progressively reducing noise and artifacts in CBCT scans through a controlled diffusion process.Results.The proposed method demonstrated superior performance in synthesizing DECT images (High DECT MAE 39.582 ± 0.855 and Low DECT MAE 48.540± 1.833) with significantly enhanced signal-to-noise ratio and reduced artifacts compared to traditional GAN-based methods. It showed marked improvements in tissue characterization and anatomical structure similarity, critical for precise proton and radiation therapy planning.Conclusions.This research has opened a new avenue in CBCT-CT synthesis for ART/APT by generating DECT images using an enhanced DDPM approach. The demonstrated similarity between the synthesized DECT images and ground truth images suggests that these synthetic volumes can be used for accurate dose calculations, leading to better adaptation in treatment planning.

Diff-MTS: Temporal-Augmented Conditional Diffusion-Based AIGC for Industrial Time Series Toward the Large Model Era.

Industrial multivariate time series (MTS) is a critical view of the industrial field for people to understand the state of machines. However, due to data collection difficulty and privacy concerns, available data for building industrial intelligence and industrial large models is far from sufficient. Therefore, industrial time series data generation is of great importance. Existing research usually applies generative adversarial networks (GANs) to generate MTS. However, GANs suffer from the unstable training process due to the joint training of the generator and discriminator. This article proposes a temporal-augmented conditional adaptive diffusion model, termed Diff-MTS, for MTS generation. It aims to better handle the complex temporal dependencies and dynamics of MTS data. Specifically, a conditional adaptive maximum-mean discrepancy (Ada-MMD) method has been proposed for the controlled generation of MTS, which does not require a classifier to control the generation. It improves the condition consistency of the diffusion model. Moreover, a temporal decomposition reconstruction UNet (TDR-UNet) is established to capture complex temporal patterns and further improve the quality of the synthetic time series. Comprehensive experiments on the C-MAPSS and FEMTO datasets demonstrate that the proposed Diff-MTS performs substantially better in terms of diversity, fidelity, and utility compared with the GAN-based methods. These results show that Diff-MTS facilitates the generation of industrial data, contributing to intelligent maintenance and the construction of industrial large models.

Unsupervised anomaly detection using inverse generative adversarial networks

Unsupervised anomaly detection (UAD) holds considerable promise for a myriad of real-world applications related to information sciences. Recently, generative models, such as Generative Adversarial Networks (GANs), have shown their potential in addressing UAD tasks. Despite their potential, current GAN-based UAD methods frequently employ intricate network architectures. Some even necessitate solving an iterative optimization problem for each test sample, making them both time-consuming and impractical for large-scale datasets or time-sensitive industrial applications. In this work, we introduce a novel generative adversarial learning method for UAD. Diverging from existing GAN-based methods, our method incorporates a distribution converter and a distribution discriminator. The converter transforms normal data into a compact target distribution such as a truncated Gaussian while the discriminator endeavors to distinguish the transformed data from the data in the target distribution. After an adversarial training process, the learned converter becomes proficient in transforming normal data into the target distribution but cannot accurately transform anomalous samples to the target distribution, which can be easily identified by the strong discriminator. Extensive experiments on nine publicly available anomaly detection benchmark datasets and a real-world, large-scale logistics dataset demonstrate that our method achieves superior detection accuracy and lower inference time than many strong baselines. Our code is publicly available in https://github.com/xiaofeng-github/IGAN.

Feature Fusion GAN Based Virtual Staining on Plant Microscopy Images.

Virtual staining of microscopy specimens using GAN-based methods could resolve critical concerns of manual staining process as displayed in recent studies on histopathology images. However, most of these works use basic-GAN framework ignoring microscopy image characteristics and their performance were evaluated based on structural and error statistics (SSIM and PSNR) between synthetic and ground-truth without considering any color space although virtual staining deals with color transformation. Besides, major aspects of staining, like color, contrast, focus, image-realness etc. were totally ignored. However, modifications of GAN architecture for virtual staining might be suitable by incorporating microscopy image features. Further, its successful implementation need to be examined by considering various aspects of staining process. Therefore, we designed, a new feature-fusion-GAN for virtual staining followed by performance assessment by framing a state-of-the-art multi-evaluation framework that includes numerous metrics in -qualitative (based on histogram-correlation of color and brightness); quantitative (SSIM and PSNR); focus aptitude (Brenner metrics and Spectral-Moments); and influence on perception (semantic perceptual influence score). For, experimental validation cell boundaries were highlighted by two different staining reagents, Safranin-O and Toluidine-Blue-O on plant microscopy images of potato tuber. We evaluated virtually stained image quality w.r.t ground-truth in RGB and YCbCr color spaces based on defined metrics and results are found very consistent. Further, impact of feature fusion has been demonstrated. Collectively, this study could be a baseline towards guiding architectural upgrading of deep pipelines for virtual staining of diverse microscopy modalities followed by future benchmark methodology or protocols.

Uncovering the Over-Smoothing Challenge in Image Super-Resolution: Entropy-Based Quantification and Contrastive Optimization.

PSNR-oriented models are a critical class of super-resolution models with applications across various fields. However, these models tend to generate over-smoothed images, a problem that has been analyzed previously from the perspectives of models or loss functions, but without taking into account the impact of data properties. In this paper, we present a novel phenomenon that we term the center-oriented optimization (COO) problem, where a model's output converges towards the center point of similar high-resolution images, rather than towards the ground truth. We demonstrate that the strength of this problem is related to the uncertainty of data, which we quantify using entropy. We prove that as the entropy of high-resolution images increases, their center point will move further away from the clean image distribution, and the model will generate over-smoothed images. Implicitly optimizing the COO problem, perceptual-driven approaches such as perceptual loss, model structure optimization, or GAN-based methods can be viewed. We propose an explicit solution to the COO problem, called Detail Enhanced Contrastive Loss (DECLoss). DECLoss utilizes the clustering property of contrastive learning to directly reduce the variance of the potential high-resolution distribution and thereby decrease the entropy. We evaluate DECLoss on multiple super-resolution benchmarks and demonstrate that it improves the perceptual quality of PSNR-oriented models. Moreover, when applied to GAN-based methods, such as RaGAN, DECLoss helps to achieve state-of-the-art performance, such as 0.093 LPIPS with 24.51 PSNR on 4× downsampled Urban100, validating the effectiveness and generalization of our approach.

BEForeGAN: An image-based deep generative approach for day-ahead forecasting of building HVAC energy consumption

This study presents a pioneering approach in building energy forecasting by introducing a novel reformulation framework that transforms the forecasting task into an image inpainting problem. Based upon the fundamental notion that “forecasting is about generating data of the future”, we propose BEForeGAN, an innovative deep generative approach for day-ahead Building HVAC Energy consumption Forecasting based on multi-channel conditional Generative Adversarial Networks (GANs) with U-Net generators. Our method is evaluated using 96,360 hourly HVAC energy consumption records from 11 buildings, demonstrating significant accuracy improvements of 17%∼76% and a substantial variability reduction of 3%∼96% compared to a suite of conventional and deep learning benchmark models across individual-building and zero-shot cross-building forecasting tasks. Notably, BEForeGAN exhibits robustness to noisy inputs, with an increase below 3% in Coefficient of Variation of Root Mean Square Error (CV-RMSE) for each 10% noise increment. This study addresses critical gaps in existing literature by showcasing the untapped potential of GANs as standalone forecasters, advocating for further exploration of two-dimensional (2D) GAN-based methods in building energy forecasting, and emphasising the need for more studies focusing on cross-building forecasting tasks. In conclusion, our findings underscore the transformative impact of GANs in revolutionising building energy forecasting practices, paving the way for enhanced energy-efficient building management and beyond.

A medical image classification method based on self-regularized adversarial learning.

Deep learning (DL) techniques have been extensively applied in medical image classification. The unique characteristics of medical imaging data present challenges, including small labeled datasets, severely imbalanced class distribution, and significant variations in imaging quality. Recently, generative adversarial network (GAN)-based classification methods have gained attention for their ability to enhance classification accuracy by incorporating realistic GAN-generated images as data augmentation. However, the performance of these GAN-based methods often relies on high-quality generated images, while large amounts of training data are required to train GAN models to achieve optimalperformance. In this study, we propose an adversarial learning-based classification framework to achieve better classification performance. Innovatively, GAN models are employed as supplementary regularization terms to support classification, aiming to address the challenges describedabove. The proposed classification framework, GAN-DL, consists of a feature extraction network (F-Net), a classifier, and two adversarial networks, specifically a reconstruction network (R-Net) and a discriminator network (D-Net). The F-Net extracts features from input images, and the classifier uses these features for classification tasks. R-Net and D-Net have been designed following the GAN architecture. R-Net employs the extracted feature to reconstruct the original images, while D-Net is tasked with the discrimination between the reconstructed image and the original images. An iterative adversarial learning strategy is designed to guide model training by incorporating multiple network-specific loss functions. These loss functions, serving as supplementary regularization, are automatically derived during the reconstruction process and require no additional dataannotation. To verify the model's effectiveness, we performed experiments on two datasets, including a COVID-19 dataset with 13958 chest x-ray images and an oropharyngeal squamous cell carcinoma (OPSCC) dataset with 3255 positron emission tomography images. Thirteen classic DL-based classification methods were implemented on the same datasets for comparison. Performance metrics included precision, sensitivity, specificity, and -score. In addition, we conducted ablation studies to assess the effects of various factors on model performance, including the network depth of F-Net, training image size, training dataset size, and loss function design. Our method achieved superior performance than all comparative methods. On the COVID-19 dataset, our method achieved , , , and in terms of precision, sensitivity, specificity, and -score, respectively. It achieved across all these metrics on the OPSCC dataset. The study to investigate the effects of two adversarial networks highlights the crucial role of D-Net in improving model performance. Ablation studies further provide an in-depth understanding of ourmethodology. Our adversarial-based classification framework leverages GAN-based adversarial networks and an iterative adversarial learning strategy to harness supplementary regularization during training. This design significantly enhances classification accuracy and mitigates overfitting issues in medical image datasets. Moreover, its modular design not only demonstrates flexibility but also indicates its potential applicability to various clinical contexts and medical imaging applications.

Generation diffusion degradation: Simple and efficient design for blind super-resolution

In the domain of computer vision, blind super-resolution is a key area focused on generating high-resolution images with enhanced visual quality from low-resolution counterparts affected by indeterminate degradation factors. This area is primarily advanced through self-supervised learning techniques utilizing GANs. Despite their prominence, GAN-based methods encounter challenges including unstable training dynamics and limited diversity, compounded by the intricate necessity to configure degradation models to mimic various blur effects and noise types. Lately, denoising diffusion models have shown promising results in image restoration, yet their sampling efficiency constraints impede their deployment in real-time scenarios. This study introduces the Generation Diffusion Degradation (GDD) model, a novel and efficient technique for replicating image degradation by applying random Gaussian noise in a sequential manner via a parametric Markov chain, followed by a progressive reconstruction of the initial image through a U-net-based noise predictor. This method adeptly mirrors the inherent degradation distribution observed in actual degraded images. Furthermore, we present an innovative training strategy that utilizes a composite loss function to train the GDD model, ensuring stable training, improving the authenticity of the generated degraded images, and precisely reflecting the degradation patterns of target images. Extensive experimental analyses underscore the superior performance of the proposed GDD model, both in objective metrics and subjective visual quality. The code is available at https://github.com/lgylab/GDD.

Conditional Diffusion Model for Urban Morphology Prediction

Predicting urban morphology based on local attributes is an important issue in urban science research. The deep generative models represented by generative adversarial network (GAN) models have achieved impressive results in this area. However, in such methods, the urban morphology is assumed to follow a specific probability distribution and be able to directly approximate the distribution via GAN models, which is not a realistic strategy. As demonstrated by the score-based model, a better strategy is to learn the gradient of the probability distribution and implicitly approximate the distribution. Therefore, in this paper, an urban morphology prediction method based on the conditional diffusion model is proposed. Implementing this approach results in the decomposition of the attribute-based urban morphology prediction task into two subproblems: estimating the gradient of the conditional distribution, and gradient-based sampling. During the training stage, the gradient of the conditional distribution is approximated by using a conditional diffusion model to predict the noise added to the original urban morphology. In the generation stage, the corresponding conditional distribution is parameterized based on the noise predicted by the conditional diffusion model, and the final prediction result is generated through iterative sampling. The experimental results showed that compared with GAN-based methods, our method demonstrated improvements of 5.5%, 5.9%, and 13.2% in the metrics of low-level pixel features, shallow structural features, and deep structural features, respectively.

A time-saving fault diagnosis using simplified fast GAN and triple-type data transfer learning

Existing intelligent fault diagnosis approaches demand substantial data for training diagnostic models. However, factors such as the inherent characteristics of bearings, operating conditions, and privacy security make collecting comprehensive fault-bearing data very difficult. Although generating synthetic data through generative adversarial networks (GANs) is feasible, the data generation of GANs is a time-consuming process. To address these challenges, a fault diagnosis framework based on GAN and deep transfer learning (DTL) is proposed, termed the simplified fast GAN triple-type data transfer learning (SFGAN-TDTL) method. Initially, an SFGAN is proposed as a replacement for traditional GANs. The time-frequency image data generated by SFGAN serve to augment the training dataset, offering faster and higher-quality data generation compared to traditional GANs. To further reduce time consumption for GAN-based methods, the TDTL method is proposed. Differing from DTL, which utilizes synthetic data to construct a pre-trained model and conducts targeted fine-tuning with real data, TDTL employs open-source data, synthetic data, and real data to fill the weights of the task-insensitive layer, task-sensitive layer, and fully connected layer, respectively. Numerical results demonstrate that SFGAN-TDTL maintains higher diagnostic accuracy while significantly reducing time consumption.

StructDiffusion: End-to-end intelligent shear wall structure layout generation and analysis using diffusion model

Shear wall structures are widely used in high-rise buildings. However, the design process of shear wall structures suffers from inefficiencies. This paper introduces StructDiffusion, an end-to-end intelligent system for shear wall layout generation and analysis. The proposed approach tackles the layout design task by employing a novel diffusion model architecture to generate conditional images. Key components of the proposed method include pretrained diffusion models, ControlNet for conditional control, and LoRA for efficient adaptation. Notably, the method allows for architectural plan images and basic textual design conditions as inputs, enabling manipulation of the generated layouts by adjusting attributes such as building height and seismic intensity. To evaluate the quality of the model's design swiftly, this paper presents a comprehensive evaluation framework that incorporates perceptual and structural validity metrics. Through experimental analyses, this paper demonstrates the effectiveness of the proposed model in generating layouts, surpassing the capabilities of GAN-based methods. Furthermore, our study investigates model-specific parameter configuration, text-guided capabilities, code compliance, and computational efficiency. StructDiffusion significantly enhances the automation and intelligence of structural engineering workflows. The framework effectively addresses challenges associated with instability, data scarcity, and limitations in assessment. This pioneering application of diffusion models opens up promising avenues for advancing data-driven structural design.

DiffGAN: An adversarial diffusion model with local transformer for MRI reconstruction

Magnetic resonance imaging (MRI) is non-invasive and crucial for clinical diagnosis, but it has long acquisition time and aliasing artifacts. Accelerated imaging techniques can effectively reduce the scanning time of MRI, thereby decreasing the anxiety and discomfort of patients. Vision Transformer (ViT) based methods have greatly improved MRI image reconstruction, but their computational complexity and memory requirements for the self-attention mechanism grow quadratically with image resolution, which limits their use for high resolution images. In addition, the current generative adversarial networks in MRI reconstruction are difficult to train stably. To address these problems, we propose a Local Vision Transformer (LVT) based adversarial Diffusion model (Diff-GAN) for accelerating MRI reconstruction. We employ a generative adversarial network (GAN) as the reverse diffusion model to enable large diffusion steps. In the forward diffusion module, we use a diffusion process to generate Gaussian mixture distribution noise, which mitigates the gradient vanishing issue in GAN training. This network leverages the LVT module with the local self-attention, which can capture high-quality local features and detailed information. We evaluate our method on four datasets: IXI, MICCAI 2013, MRNet and FastMRI, and demonstrate that Diff-GAN can outperform several state-of-the-art GAN-based methods for MRI reconstruction.

STGlow: A Flow-Based Generative Framework With Dual-Graphormer for Pedestrian Trajectory Prediction.

The pedestrian trajectory prediction task is an essential component of intelligent systems. Its applications include but are not limited to autonomous driving, robot navigation, and anomaly detection of monitoring systems. Due to the diversity of motion behaviors and the complex social interactions among pedestrians, accurately forecasting their future trajectory is challenging. Existing approaches commonly adopt generative adversarial networks (GANs) or conditional variational autoencoders (CVAEs) to generate diverse trajectories. However, GAN-based methods do not directly model data in a latent space, which may make them fail to have full support over the underlying data distribution. CVAE-based methods optimize a lower bound on the log-likelihood of observations, which may cause the learned distribution to deviate from the underlying distribution. The above limitations make existing approaches often generate highly biased or inaccurate trajectories. In this article, we propose a novel generative flow-based framework with a dual-graphormer for pedestrian trajectory prediction (STGlow). Different from previous approaches, our method can more precisely model the underlying data distribution by optimizing the exact log-likelihood of motion behaviors. Besides, our method has clear physical meanings for simulating the evolution of human motion behaviors. The forward process of the flow gradually degrades complex motion behavior into simple behavior, while its reverse process represents the evolution of simple behavior into complex motion behavior. Furthermore, we introduce a dual-graphormer combined with the graph structure to more adequately model the temporal dependencies and the mutual spatial interactions. Experimental results on several benchmarks demonstrate that our method achieves much better performance compared to previous state-of-the-art approaches.

AnimeDiffusion: Anime Diffusion Colorization.

Being essential in animation creation, colorizing anime line drawings is usually a tedious and time-consuming manual task. Reference-based line drawing colorization provides an intuitive way to automatically colorize target line drawings using reference images. The prevailing approaches are based on generative adversarial networks (GANs), yet these methods still cannot generate high-quality results comparable to manually-colored ones. In this article, a new AnimeDiffusion approach is proposed via hybrid diffusions for the automatic colorization of anime face line drawings. This is the first attempt to utilize the diffusion model for reference-based colorization, which demands a high level of control over the image synthesis process. To do so, a hybrid end-to-end training strategy is designed, including phase 1 for training diffusion model with classifier-free guidance and phase 2 for efficiently updating color tone with a target reference colored image. The model learns denoising and structure-capturing ability in phase 1, and in phase 2, the model learns more accurate color information. Utilizing our hybrid training strategy, the network convergence speed is accelerated, and the colorization performance is improved. Our AnimeDiffusion generates colorization results with semantic correspondence and color consistency. In addition, the model has a certain generalization performance for line drawings of different line styles. To train and evaluate colorization methods, an anime face line drawing colorization benchmark dataset, containing 31,696 training data and 579 testing data, is introduced and shared. Extensive experiments and user studies have demonstrated that our proposed AnimeDiffusion outperforms state-of-the-art GAN-based methods and another diffusion-based model, both quantitatively and qualitatively.

Biomarkers-Aware Asymmetric Bibranch GAN with Adaptive Memory Batch Normalization for Prediction of Anti-VEGF Treatment Response in Neovascular Age-related Macular Degeneration.

The emergence of anti-vascular endothelial growth factor (anti-VEGF) therapy has revolutionized neovascular age-related macular degeneration (nAMD). Post-therapeutic optical coherence tomography (OCT) imaging facilitates the prediction of therapeutic response to anti-VEGF therapy for nAMD. Although the generative adversarial network (GAN) is a popular generative model for post-therapeutic OCT image generation, it is realistically challenging to gather sufficient pre- and post-therapeutic OCT image pairs, resulting in overfitting. Moreover, the available GAN-based methods ignore local details, such as the biomarkers that are essential for nAMD treatment. To address these issues, a Biomarkers-aware Asymmetric Bibranch GAN (BAABGAN) is proposed to efficiently generate post-therapeutic OCT images. Specifically, one branch is developed to learn prior knowledge with a high degree of transferability from large-scale data, termed the source branch. Then, the source branch transfer knowledge to another branch, which is trained on small-scale paired data, termed the target branch. To boost the transferability, a novel Adaptive Memory Batch Normalization (AMBN) is introduced in the source branch, which learns more effective global knowledge that is impervious to noise via memory mechanism. Also, a novel Adaptive Biomarkers-aware Attention (ABA) module is proposed to encode biomarkers information into latent features of target branches to learn finer local details of biomarkers. The proposed method outperforms traditional GAN models and can produce high-quality post-treatment OCT pictures with limited data sets, as shown by the results of experiments.

Leveraging Ensemble Learning with Generative Adversarial Networks for Imbalanced Software Defects Prediction

Software defect prediction is an active research area. Researchers have proposed many approaches to overcome the imbalanced defect problem and build highly effective machine learning models that are not biased towards the majority class. Generative adversarial networks (GAN) are one of the state-of-the-art techniques that can be used to generate synthetic samples of the minority class and produce a balanced dataset. However, it was not investigated thoroughly in the area of imbalanced defect prediction. In this paper, we proposed to combine GAN-based methods with boosting ensembles to yield robust defect prediction models. GAN-based methods were used to balance the defect datasets, and the AdaBoost ensemble was employed to classify the modules into defective and non-defective modules. Our proposed approach was investigated within the context of 10 software defect datasets with different imbalance ratios. Wilcoxon effect size and Scott–Knott effect size difference tests were used as statistical tests to quantify the model’s performance differences statistically. Empirical results indicated that GAN-based methods need hyperparameter optimization when used for imbalanced software defect prediction. In comparison to the traditional sampling techniques, GAN methods outperformed all traditional techniques when used for imbalanced defect prediction. Lastly, results demonstrated that GAN-based methods should not be combined with undersampling to handle imbalance problems.

Synthetic demand data generation for individual electricity consumers: Inpainting

In this contribution we deal with the problem of producing “reasonable” data, when considering recorded energy consumption data, which are at certain sections incomplete and/or erroneous. This task is important, when energy providers employ prediction models for expected energy consumption, which are based on past recorded consumption data, which then of course should be reliable and valid. In a related contribution Yilmaz (2022), GAN-based methods for producing such “artificial data” have been investigated. In this contribution, we describe an alternative and complementary method based on signal inpainting, which has been successfully applied to audio processing Lieb and Stark (2018). After giving a short overview of the theory of proximity-based convex optimization, we describe and adapt an iterative inpainting scheme to our problem. The usefulness of this approach is demonstrated by analyzing real-world-data provided by a German energy supplier.

Two-branch crisscross network for realistic and accurate image super-resolution

Convolutional Neural Networks have made remarkable progress in single-image super-resolution. However, existing methods struggle to balance reconstruction accuracy and perceptual quality, resulting in unsatisfactory outcomes. To address this challenge, we propose the Two-Branch Crisscross Generative Adversarial Network (TBCGAN) for achieving accurate and realistic super-resolution results. TBCGAN employs two asymmetric branches that separately reconstruct high-frequency (HF) and low-frequency (LF) images, leveraging their distinct information and reconstruction requirements. To ensure coherent results, we apply different supervision to the reconstructed HF, LF, and super-resolution (SR) images while facilitating information interaction through the interleaving and fusion of HF and LF features. Extensive experimental evaluations demonstrate that TBCGAN achieves an excellent balance between reconstruction accuracy and perceptual quality, outperforming GAN-based methods in reconstruction accuracy and MSE-based methods in perceptual quality.

Multi-image transformer for multi-focus image fusion

Multi-Focus Image Fusion (MFIF) is an image enhancement task that fuses images in which different regions are in focus to achieve an all-in-focus image. In recent years, Generative Adversarial Networks (GANs)-based approaches have significantly improved the MFIF on Convolutional Neural Network (CNN) architectures. However, despite vision transformers (ViTs) achieving more successful results than CNNs in many high and low-level vision problems due to their ability to provide global connectivity, they have not been employed for MFIF until this study. We develop a Multi-image Transformer (MiT) for MFIF by being inspired by a Spatial-Temporal Transformer Network (STTN) so that global connection can be modeled along multiple input images. We call the proposed transformers-based MFIF model MiT-MFIF as it uses the developed MiT as a core component. We have made various modifications to the baseline transformer to be able to utilize vision transformers in MFIF tasks. Comprehensive experiments on standard MFIF datasets demonstrate the effectiveness of the proposed MiT-MFIF. Moreover, proposed method does not require any post-processing step like in competitor GAN-based methods while outperforming the state-of-the-art MFIF methods.

Hairstyle transfer via manipulating decoupled latent codes of StyleGAN2

Hairstyle transfer aims to combine the hairstyle attributes of the reference images with the face and background of the identity image into one realistic image. This is challenging due to the differences in semantic structure between reference images and identity image, as well as the tangled hairstyle attributes. Existing GAN-based methods tend to simply combine image patches of hair, face, and background, which leads to discordant semantic structures and obvious artifacts in the results. And the GAN inversion-based methods require elaborate orthogonalization of loss gradients with additional calculations to alleviate the entanglement between the complex hairstyle attributes. To address these limitations, we propose a novel framework for hairstyle transfer based on StyleGAN inversion. Specifically, instead of simply combining the image patches, we use StyleGAN’s generative power to get more reasonable target semantic masks. During the alignment with the target masks, background inpainting is performed by StyleGAN rather than auxiliary models, which makes the results more realistic. Furthermore, our method decomposes hairstyle into three attributes: structure, texture, and color to better control the hairstyle transfer process. Besides, we observe that the spatial scales of these attributes match those of the StyleGAN’s latent codes. Then, we decouple the latent code in the hierarchical dimension and use different layers of the latent code to complete the transfer of different hair attributes, which reduces the entanglement between attributes without extra calculations. Correspondingly, our method divides hairstyle transfer into three steps, making the transfer process more controllable. By comparing with state-of-the-art methods, we verify the effectiveness of our method.