Joint face completion and super-resolution using multi-scale feature relation learning

Abstract

Previous research on face restoration often focused on repairing specific types of low-quality facial images such as low-resolution (LR) or occluded facial images. However, in the real world, both the above-mentioned forms of image degradation often coexist. Therefore, it is important to design a model that can repair images that are LR and occluded simultaneously. This paper proposes a multi-scale feature graph generative adversarial network (MFG-GAN) to carry out face restoration in contexts in which both LR and occluded degradation modes coexist, and also to repair images with a single type of degradation. Based on the GAN, the MFG-GAN integrates the graph convolution and feature pyramid networks to restore occluded low-resolution face images to non-occluded high-resolution face images. The MFG-GAN uses a set of customized losses to ensure that high-quality images are generated. In addition, we designed the network in an end-to-end format. We conduct experiments on general face image restoration and facial expression restoration. Experimental results on the public-domain databases show that the proposed approach outperforms state-of-the-art methods in performing face super resolution (up to 4× or 8×) and face completion simultaneously and can recover better facial expression details.