A Structure-Preserving and Illumination-Consistent Cycle Framework for Image Harmonization

Abstract

In a composite image, the foreground and background are filmed under different scenarios, such as different lighting conditions, causing inconsistency and reducing the overall realism of the image. Image harmonization aims to generate visually realistic composite images by adjusting the foreground to the background conditions while maintaining the structure. Existing methods focus on adjusting the foreground object by directly training the foreground generation network with the ground truth, neglecting the different roles of the illumination and structure of the foreground in image harmonization. Moreover, the use of background, except for providing illumination, is not thoroughly investigated in this task. In this paper, we propose a structure-preserving and illumination-consistent cycle (SP-IC cycle) framework for image harmonization by exploring the illumination and structure of both the foreground and background. It achieves image harmonization by specifically changing the illumination and keeping the structure instead of ambiguously changing the foreground. Then, an illumination-consistent foreground harmonization cycle is developed to change the foreground illumination, while a structure-preserving cycle is designed to keep the foreground structure. Background information is explored in both cycles to assist in decomposing the illumination and structure of the foreground. In addition, the proposed SP-IC cycle framework can be applied to any image harmonization method to further boost its performance. Experimental results demonstrate that our method achieves better harmonious image quality than state-of-the-art methods, especially on an illumination-varying dataset.