Diverse image inpainting with disentangled uncertainty

Abstract

Most existing inpainting methods repair a corrupted image to a single output, which gives people no choice to select the most satisfactory result. However, image inpainting is essentially a multi-modal problem because the inpainted results could have multiple possibilities. To generate both diverse and realistic inpainted results, we propose a diverse image inpainting framework with disentangled uncertainty. We disentangle the uncertainty of the missing region into two aspects: structure and appearance. Correspondingly, we divide the process of diverse image inpainting into two stages: diverse structure inpainting and diverse appearance inpainting. In the first stage, we restore the structure of the missing region, producing diverse complete edge maps. In the second stage, using a complete edge map as the guidance, we fill in diverse appearance information of the missing region. We also design a light-weighted disentangling subnetwork to disentangle structure information and appearance information. Besides, we propose a novel style-based masked residual block to better deal with the uncertainty. Experiments on CelebA-HQ, Paris Street View, and Places2 demonstrate that our method can repair the corrupted image with higher fidelity and diversity than other existing methods.