Mask-guided cycle-GAN for specular highlight removal

Abstract

• A novel Cycle-GAN for specular highlight removal based on unpaired data. • The network is guided by a highlight mask to focus on highlight regions. • The training is divided into two steps through a luminance guidance structure. • The proposed method obtained high-quality highlight-free images even on unpaired natural images Specular highlight removal is an important yet challenging problem in image enhancement. Recent methods based on deep learning and trained by massive paired or unpaired data have demonstrated promising performance for this task. Methods based on unpaired data have recently gained more attention for easier training data collection. In this paper, we present a Mask-Guided Cycle-GAN for specular highlight removal on unpaired data. Incorporating the idea that specular highlight mainly has characteristics in lightness, we attempt to train a module only on luminance channel before considering all channels, and then adopt the training results to guide the subsequent highlight removal module. We further convert the highlight removal problem to image-to-image translation by using cycle-consistent adversarial network (Cycle-GAN). In the proposed network, a non-negative matrix factorization (NMF) based method is incorporated to obtain accurate highlight masks. The proposed method is evaluated using the specular highlight image quadruples (SHIQ) and the LIME datasets, and the advantages are demonstrated via comparative experimental results.