Advancing generalizations of multi-scale GAN via adversarial perturbation augmentations

Abstract

The multi-scale generative adversarial networks (MS-GANs) typically introduce a series of GANs to process hierarchical image patches from a single image, which suites image manipulation scenarios where it takes one image as input to produce manipulated results. However, during MS-GAN training, leveraging only a single image in hierarchical patches brings data insufficiency for model generalization, which limits MS-GAN to produce fine-grained texture details. To address this problem, we theoretically formulate MS-GAN generalization from the PAC-Bayes bound perspective in this work. Under this formulation, we obtain a non-vacuous upper bound of the generalization error, and propose an adversarial perturbation augmentation method named AP-Aug. Given an input image, our AP-Aug produces adversarial perturbations to augment the original image. The augmented training images benefit MS-GAN to approach the generalization upper bound. To this end, we advance the generalizations of MS-GAN to improve image generation performance. In the experiments, our AP-Aug benefits MS-GAN to achieve high quality image generation under several image manipulation scenarios including paint2image, image style transfer, and image super-resolution. The improvement shows that our AP-Aug advances the MS-GAN generalizations for high quality image generations, which perform favorably against state-of-the-art approaches.