Multistage attention network for image inpainting

Abstract

Image inpainting refers to the process of restoring the mask regions of damaged images. Existing inpainting algorithms have exhibited outstanding performance on certain inpainting tasks that are focused on recovering small masks or square masks. Tasks that attempt to reconstruct large proportion of damaged images can still be improved. Although many attention-related algorithms have been proposed to solve image inpainting tasks, most of them ignore the requirements to balancing the detail and style level. In this paper, we propose a novel image inpainting method for large-scale irregular masks. We introduce a special multistage attention module that considers structure consistency and detail fineness. The proposed multistage attention module operates in a coarse to-fine manner, where the early stage performs large feature patch swapping and ensures the global consistency in images, and the next stage swaps small patches to refine the texture. Then, we adopt a partial convolution strategy to avoid the misuse of invalid data during convolution. Several losses are combined as the training objective function to generate excellent results with global consistency and exquisite detail. Qualitative and quantitative experiments on the Paris StreetView, CelebA, and Places2 datasets demonstrate the superior performance of the proposed approach compared with state-of-the-art models.