A text guided cross modal joint inpainting algorithm for ancient murals

Abstract

Existing deep learning algorithms for ancient mural inpainting only consider prior information of mural images to guide the inpainting of damaged areas, lacking guidance from textual semantic information, which often leads to problems such as inconsistent semantics and disordered textures in the inpainting results. In this work, we propose a text guided cross modal joint inpainting algorithm for ancient murals. Firstly, a cross modal joint inpainting network for ancient murals was constructed, which introduced text semantic information as the control guidance for mural inpainting, providing contextual information for mural image inpainting and guiding the inpainting of damaged areas. Then, design a text encoder and an image encoder to extract text and image features respectively, including word and sentence features. Input the sentence features into the conditional enhancement module, resample the text sentences, and solve the problem of text sparsity. Secondly, design a text decoupling module that utilizes a multimodal attention mechanism to extract the semantic information of the missing parts of damaged murals, and design a text semantic enhancement module to improve the semantic representation ability of the text, achieve the repair of local details of murals, and solve the problem of texture disorder. Finally, the reconstruction and repair of murals are completed through a residual discriminator game and a spectral normalized Markov dual discriminator against the mural generation network. Through inpainting experiments on real Dunhuang murals, it has been shown that the proposed method is more effective than comparative methods in repairing damaged murals, achieving better visual perception and coordination in reconstruction, and outperforming comparative algorithms in both subjective and objective evaluations.