Semantic prior-driven fused contextual transformation network for image inpainting

Abstract

Recent advances in image inpainting have achieved impressive performance for generating plausible visual details on small regular image defects or simple backgrounds. However, current solution suffers from the lack of semantic priors for the image and the inability to deduce the image content from distant background, leading to distorted structures and artifacts in the results when inpainting large random irregular complicated images. To address these problems, a semantic prior-driven fused contextual transformation network for image inpainting is proposed as a promise solution. First, the semantic prior generator is put forward to map the semantic features of ground truth images and the low-level features of broken images to semantic priors. Subsequently, an image split-transform-aggregated strategy, named fusion context transformation block, is presented to infer rich multi-scale remote texture features and thus to improve the restored image finesse. Thereafter, an aggregated semantic attention-aware module, consisting of spatially adaptive normalization and enhanced spatial attention is designed to aggregate semantic priors and multi-scale texture features into the decoder to restore reasonable structure. Finally, the mask guided discriminator is developed to effectively discriminate between real and false pixels in the output image to improve the capability of the discriminator and hence to reduce the probability of artifacts containing in the output image. Comprehensive experimental results on CelebA-HQ, Paris Street View, and Places2 datasets demonstrate the superiority of the proposed network over the state-of-the-arts, whose PSNR, SSIM and MAE are improved about 20 %, 12.6 %, and 42 % gains, respectively.