Fully automatic image colorization based on semantic segmentation technology.

Min Xu,Youdong Ding

doi:10.1371/journal.pone.0259953

Abstract

Aiming at these problems of image colorization algorithms based on deep learning, such as color bleeding and insufficient color, this paper converts the study of image colorization to the optimization of image semantic segmentation, and proposes a fully automatic image colorization model based on semantic segmentation technology. Firstly, we use the encoder as the local feature extraction network and use VGG-16 as the global feature extraction network. These two parts do not interfere with each other, but they share the low-level feature. Then, the first fusion module is constructed to merge local features and global features, and the fusion results are input into semantic segmentation network and color prediction network respectively. Finally, the color prediction network obtains the semantic segmentation information of the image through the second fusion module, and predicts the chrominance of the image based on it. Through several sets of experiments, it is proved that the performance of our model becomes stronger and stronger under the nourishment of the data. Even in some complex scenes, our model can predict reasonable colors and color correctly, and the output effect is very real and natural.

Highlights

Many fields, including old photo and old movie restoration, remote sensing image, and biologic medical image, have strong demand for image colorization technology
Through comparing the coloring effects of eight groups of test images, we find that the images processed by three models proposed in 2016 have obvious color bleeding, the images processed by Lei et al [47] have simple colors and Su et al [48] have slight color overflow, while our algorithm with the high-level semantic segmentation information of the image itself has strong robustness, which can apply to natural image colorization in different scenes
Since color images have incomparable advantages over black-and-white images in terms of people’s visual perception and subsequent image understanding and analysis, it is of great significance to continue to study a practical grayscale image colorization algorithm

Summary

Introduction

Many fields, including old photo and old movie restoration, remote sensing image, and biologic medical image, have strong demand for image colorization technology. The goal of image colorization is to assign colors to each pixel of a grayscale image, and the researches on this subject has been in the ascendant. The earliest research on this subject was Markle [1], who realized the colorization of the moon image with the help of computer aided technologies, which attracted wide attention from all walks of life. It is prone to problems such as color bleeding and boundary blurring when dealing with images with complex textures.

Methods

Results

Conclusion